2024-04-15T22:38:46.619 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T22:38:46.620 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_teuthology@teuthology --name teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608 --verbose -- /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608/orig.config.yaml 2024-04-15T22:38:46.672 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T22:38:46.784 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7657608' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.0 ' name: teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 7775 sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 sleep_before_teardown: 0 subset: 18/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLZmM5gvh1qzwCEtC1kOi+EYXitmVn46BT27K1/Vstizybv9ZrD4lUjcg3tHtS5YT1WLD+qu0mAE3oovDFJN5I= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP2CqhUmieuRAYfPqi7ZKgTMYFB2yekVdQVGUOjtELRJVw4kwAKHMbd0EWWVYnJxrFzXmmC2+2AAqANYA3B/FHc= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.0 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.0 roleless: true - print: '**** done end installing v18.2.0 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /home/teuthology/src/git.ceph.com_ceph_squid/qa/suites/orch/cephadm/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-15T22:38:46.784 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa; will attempt to use it 2024-04-15T22:38:46.785 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks 2024-04-15T22:38:46.785 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T22:38:46.787 INFO:teuthology.task.internal:Checking packages... 2024-04-15T22:38:46.810 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a9a752df26c63acad72e1b3569fd79a515ca0765' 2024-04-15T22:38:46.811 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T22:38:46.811 INFO:teuthology.packaging:ref: None 2024-04-15T22:38:46.811 INFO:teuthology.packaging:tag: None 2024-04-15T22:38:46.811 INFO:teuthology.packaging:branch: squid 2024-04-15T22:38:46.811 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:38:46.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-15T22:38:46.989 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2114.ga9a752df 2024-04-15T22:38:46.991 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T22:38:46.997 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T22:38:46.997 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T22:38:47.042 INFO:teuthology.task.internal:Saving configuration 2024-04-15T22:38:47.061 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T22:38:47.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:38:47.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:32:20.117607', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLZmM5gvh1qzwCEtC1kOi+EYXitmVn46BT27K1/Vstizybv9ZrD4lUjcg3tHtS5YT1WLD+qu0mAE3oovDFJN5I='} 2024-04-15T22:38:47.112 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:32:20.119725', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP2CqhUmieuRAYfPqi7ZKgTMYFB2yekVdQVGUOjtELRJVw4kwAKHMbd0EWWVYnJxrFzXmmC2+2AAqANYA3B/FHc='} 2024-04-15T22:38:47.112 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T22:38:47.119 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-15T22:38:47.119 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-15T22:38:47.119 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T22:38:47.203 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbccbcf40d0>, signals=[15]) 2024-04-15T22:38:47.204 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T22:38:47.210 INFO:teuthology.task.internal:Opening connections... 2024-04-15T22:38:47.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:38:47.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:38:47.319 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-04-15T22:38:47.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:38:47.409 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T22:38:47.415 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-04-15T22:38:47.432 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-04-15T22:38:47.432 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T22:38:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-04-15T22:38:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-04-15T22:38:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T22:38:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-04-15T22:38:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T22:38:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T22:38:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T22:38:47.490 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-04-15T22:38:47.510 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-04-15T22:38:47.527 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-04-15T22:38:47.527 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-04-15T22:38:47.583 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T22:38:47.584 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T22:38:47.585 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T22:38:47.585 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-04-15T22:38:47.601 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T22:38:47.610 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T22:38:47.634 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T22:38:47.634 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-04-15T22:38:47.637 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-04-15T22:38:47.652 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T22:38:47.658 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T22:38:47.658 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-04-15T22:38:47.695 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-04-15T22:38:47.835 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T22:38:47.897 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T22:38:47.912 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-15T22:38:47.912 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-15T22:38:47.912 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T22:38:47.913 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T22:38:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:38:47.929 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-15T22:38:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:38:47.930 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-04-15T22:38:47.985 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-361.el9.x86_64 2024-04-15T22:38:47.985 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-361.el9.x86_64 2024-04-15T22:38:47.986 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-04-15T22:38:47.987 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-361.el9.x86_64 2024-04-15T22:38:47.987 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.14.0-361.el9.x86_64 2024-04-15T22:38:47.987 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-04-15T22:38:50.137 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-15T22:38:50.223 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-15T22:38:51.715 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-04-15T22:38:51.762 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-04-15T22:38:52.797 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-15T22:38:55.015 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-15T22:38:57.030 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:04 2024-04-15T22:38:59.323 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-15T22:38:59.448 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.2 MB 00:00 2024-04-15T22:39:01.014 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-15T22:39:02.540 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-15T22:39:05.257 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.2 MB 00:02 2024-04-15T22:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-15T22:39:07.590 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-15T22:39:08.209 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-15T22:39:09.988 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T22:39:09.989 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T22:39:10.084 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:39:10.094 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-04-15T22:39:10.095 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:10.096 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 77 M 2024-04-15T22:39:10.096 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 123 M 2024-04-15T22:39:10.096 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:39:10.399 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 913 kB/s | 277 kB 00:00 2024-04-15T22:39:11.025 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-04-15T22:39:11.633 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-04-15T22:39:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-15T22:39:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:39:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:Total 40 MB/s | 77 MB 00:01 2024-04-15T22:39:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:39:12.389 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:39:12.389 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:39:13.225 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:39:13.225 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:39:13.401 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-15T22:39:15.559 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:39:15.866 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T22:39:15.866 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T22:39:15.959 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:39:15.966 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:39:15.967 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-04-15T22:39:15.967 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:39:15.967 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 77 M 2024-04-15T22:39:15.967 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 123 M 2024-04-15T22:39:15.967 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:39:16.127 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-15T22:39:16.289 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:39:16.384 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:39:16.702 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-04-15T22:39:17.203 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-04-15T22:39:17.553 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-15T22:39:17.556 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:39:17.556 INFO:teuthology.orchestra.run.smithi037.stdout:Total 48 MB/s | 77 MB 00:01 2024-04-15T22:39:17.556 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:39:18.014 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:39:18.014 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:39:18.889 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:39:18.889 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:39:19.142 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:39:19.277 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:39:21.325 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:39:21.941 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:39:22.029 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:39:23.961 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:39:24.012 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:39:25.057 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:39:28.853 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:30.149 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:39:30.190 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:35.293 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:58.212 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:58.212 INFO:teuthology.orchestra.run.smithi189.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T22:39:58.212 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:58.256 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:58.624 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:58.624 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:39:58.624 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:39:58.624 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:39:59.055 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:39:59.055 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:59.055 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:39:59.056 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T22:39:59.056 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T22:39:59.056 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T22:39:59.056 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T22:39:59.056 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:59.056 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:39:59.387 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:39:59.507 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:39:59.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:39:59.508 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T22:39:59.508 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T22:39:59.508 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T22:39:59.508 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-15T22:39:59.508 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T22:39:59.508 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:39:59.590 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:39:59.591 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:39:59.591 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T22:39:59.625 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:39:59.626 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T22:39:59.627 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T22:39:59.627 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 5.14.0-438.el9.x86_64 2024-04-15T22:39:59.627 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T22:40:00.231 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-04-15T22:40:01.561 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T22:40:01.593 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-04-15T22:40:01.596 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-04-15T22:40:01.611 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.tROLPu7QMt 2024-04-15T22:40:01.612 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tROLPu7QMt 2024-04-15T22:40:01.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.tROLPu7QMt 2024-04-15T22:40:01.820 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.tROLPu7QMt is 6KB 2024-04-15T22:40:01.869 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.tROLPu7QMt 2024-04-15T22:40:01.883 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T22:40:01.948 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T22:40:01.948 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T22:40:01.948 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T22:40:01.948 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T22:40:01.949 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T22:40:02.102 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-04-15T22:40:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:40:06.128 INFO:teuthology.orchestra.run.smithi037.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T22:40:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:40:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:40:06.519 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:40:06.520 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:40:06.520 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:40:06.520 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:40:06.911 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:40:06.912 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:40:07.243 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:40:07.351 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:40:07.352 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:40:07.352 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T22:40:07.352 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T22:40:07.352 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T22:40:07.352 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-15T22:40:07.352 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T22:40:07.352 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:40:07.435 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:40:07.436 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:40:07.436 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-04-15T22:40:07.473 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.06 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 68.el9 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Group : Unspecified 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 8264254 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T22:40:07.474 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : builder@centos.org 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:40:07.475 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T22:40:07.476 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T22:40:07.476 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 5.14.0-438.el9.x86_64 2024-04-15T22:40:07.476 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T22:40:08.073 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-04-15T22:40:09.581 INFO:teuthology.orchestra.run.smithi037.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T22:40:09.601 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-04-15T22:40:09.603 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-04-15T22:40:09.658 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.xfcalc8pMz 2024-04-15T22:40:09.659 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xfcalc8pMz 2024-04-15T22:40:09.734 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.xfcalc8pMz 2024-04-15T22:40:09.877 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.xfcalc8pMz is 6KB 2024-04-15T22:40:09.927 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.xfcalc8pMz 2024-04-15T22:40:09.942 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T22:40:10.010 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T22:40:10.010 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T22:40:10.010 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T22:40:10.010 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T22:40:10.011 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T22:40:10.147 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-04-15T22:40:32.126 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-15T22:40:32.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:40:32.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:40:40.156 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-15T22:40:40.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T22:40:40.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:40:50.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-15T22:40:58.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T22:40:59.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:40:59.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:41:02.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-15T22:41:07.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T22:41:07.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:41:14.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:41:14.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:41:26.050 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T22:41:38.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T22:41:38.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:41:38.416 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-15T22:41:38.854 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T22:41:38.854 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T22:41:38.854 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-04-15T22:41:38.868 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-438.el9.x86_64 2024-04-15T22:41:38.869 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T22:41:38.869 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T22:41:38.869 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T22:41:39.870 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-15T22:41:39.870 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T22:41:39.949 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-04-15T22:41:39.977 DEBUG:teuthology.parallel:result is None 2024-04-15T22:42:14.535 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T22:42:29.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:42:29.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:42:29.972 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-15T22:42:30.356 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:42:30.357 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T22:42:30.357 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-15T22:42:30.372 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-438.el9.x86_64 2024-04-15T22:42:30.372 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T22:42:30.372 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T22:42:30.372 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T22:42:31.373 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-15T22:42:31.374 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T22:42:31.443 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-04-15T22:42:31.481 DEBUG:teuthology.parallel:result is None 2024-04-15T22:42:31.481 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T22:42:31.489 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T22:42:31.489 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T22:42:31.492 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T22:42:31.510 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T22:42:31.532 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T22:42:31.565 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T22:42:31.565 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T22:42:31.567 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T22:42:31.600 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T22:42:31.607 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T22:42:31.607 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T22:42:31.632 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T22:42:31.667 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:42:31.677 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:42:31.682 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:42:31.693 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:42:31.694 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T22:42:31.701 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T22:42:31.701 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T22:42:31.725 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T22:42:31.765 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T22:42:31.774 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T22:42:31.774 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T22:42:31.796 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T22:42:31.822 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:42:31.895 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:42:31.947 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:42:32.030 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:42:32.074 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:42:32.074 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T22:42:32.142 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:42:32.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:42:32.252 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:42:32.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:42:32.340 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:42:32.340 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T22:42:32.409 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-04-15T22:42:32.411 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-04-15T22:42:32.456 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:42:32.493 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:42:32.862 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T22:42:32.870 INFO:teuthology.task.internal:Starting timer... 2024-04-15T22:42:32.870 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T22:42:32.910 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T22:42:32.943 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-15T22:42:32.943 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-04-15T22:42:33.017 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-04-15T22:42:33.018 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-04-15T22:42:33.071 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-04-15T22:42:33.073 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T22:42:33.073 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-04-15T22:42:33.117 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-04-15T22:42:33.117 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-04-15T22:42:33.144 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-04-15T22:42:33.144 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-04-15T22:42:33.144 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T22:42:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:42:33.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T22:42:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:42:33.216 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T22:42:33.216 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-04-15T22:42:33.278 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-04-15T22:42:33.312 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T22:42:33.322 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-15T22:42:33.327 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-15T22:42:33.447 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T22:42:33.463 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T22:42:33.464 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T22:47:05.897 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-04-15T22:47:05.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T22:47:05.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:47:06.027 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-15T22:47:06.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T22:47:06.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:47:06.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:47:06.221 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-15T22:47:06.304 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-15T22:47:06.304 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T22:47:06.316 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T22:47:06.316 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T22:47:06.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:47:06.320 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T22:47:06.320 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:47:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T22:47:06.374 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T22:47:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T22:47:06.403 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T22:47:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-04-15T22:47:06.422 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-04-15T22:47:06.435 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-04-15T22:47:06.440 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T22:47:06.450 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-04-15T22:47:06.458 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T22:47:06.468 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T22:47:06.485 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T22:47:06.511 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:47:06.517 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:47:06.517 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-04-15T22:47:06.517 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.517 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.517 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.517 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.538 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:47:06.543 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:47:06.543 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-15T22:47:06.543 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.543 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.543 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.543 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:47:06.543 INFO:teuthology.run_tasks:Running task install... 2024-04-15T22:47:06.552 DEBUG:teuthology.task.install:project ceph 2024-04-15T22:47:06.552 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'}} 2024-04-15T22:47:06.552 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.0', 'flavor': 'default', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'} 2024-04-15T22:47:06.553 INFO:teuthology.task.install:Using flavor: default 2024-04-15T22:47:06.562 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-15T22:47:06.562 INFO:teuthology.task.install:extra packages: [] 2024-04-15T22:47:06.563 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-04-15T22:47:06.563 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-15T22:47:06.563 INFO:teuthology.packaging:ref: None 2024-04-15T22:47:06.563 INFO:teuthology.packaging:tag: v18.2.0 2024-04-15T22:47:06.563 INFO:teuthology.packaging:branch: None 2024-04-15T22:47:06.563 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:47:08.714 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.0^{} -> 5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-15T22:47:08.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-15T22:47:08.718 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-04-15T22:47:08.718 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-15T22:47:08.719 INFO:teuthology.packaging:ref: None 2024-04-15T22:47:08.719 INFO:teuthology.packaging:tag: v18.2.0 2024-04-15T22:47:08.719 INFO:teuthology.packaging:branch: None 2024-04-15T22:47:08.719 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:47:08.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-15T22:47:08.874 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-04-15T22:47:08.875 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-04-15T22:47:08.913 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T22:47:08.914 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:47:08.914 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T22:47:08.948 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-15T22:47:08.948 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-15T22:47:08.948 INFO:teuthology.packaging:ref: None 2024-04-15T22:47:08.948 INFO:teuthology.packaging:tag: v18.2.0 2024-04-15T22:47:08.948 INFO:teuthology.packaging:branch: None 2024-04-15T22:47:08.948 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:47:08.949 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T22:47:08.953 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-04-15T22:47:08.953 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-04-15T22:47:08.984 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T22:47:08.984 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:47:08.984 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T22:47:09.018 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-15T22:47:09.018 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-15T22:47:09.018 INFO:teuthology.packaging:ref: None 2024-04-15T22:47:09.019 INFO:teuthology.packaging:tag: v18.2.0 2024-04-15T22:47:09.019 INFO:teuthology.packaging:branch: None 2024-04-15T22:47:09.019 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:47:09.019 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T22:47:09.021 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T22:47:09.092 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T22:47:09.109 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T22:47:09.138 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-04-15T22:47:09.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-04-15T22:47:09.182 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T22:47:09.253 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-04-15T22:47:09.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-04-15T22:47:09.395 INFO:teuthology.orchestra.run.smithi037.stdout:74 files removed 2024-04-15T22:47:09.423 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-04-15T22:47:09.528 INFO:teuthology.orchestra.run.smithi189.stdout:74 files removed 2024-04-15T22:47:09.555 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw 2024-04-15T22:47:09.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 505 kB/s | 76 kB 00:00 2024-04-15T22:47:10.003 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 528 kB/s | 76 kB 00:00 2024-04-15T22:47:10.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 74 kB/s | 9.3 kB 00:00 2024-04-15T22:47:10.171 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 78 kB/s | 9.3 kB 00:00 2024-04-15T22:47:10.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 18 kB/s | 2.2 kB 00:00 2024-04-15T22:47:10.338 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 17 kB/s | 2.2 kB 00:00 2024-04-15T22:47:11.132 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.7 MB 00:00 2024-04-15T22:47:11.144 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-04-15T22:47:12.594 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-04-15T22:47:12.674 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-04-15T22:47:13.378 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-04-15T22:47:13.733 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-15T22:47:15.228 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-15T22:47:15.658 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-15T22:47:19.690 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.2 MB 00:00 2024-04-15T22:47:20.992 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.2 MB 00:01 2024-04-15T22:47:21.463 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-15T22:47:22.593 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-04-15T22:47:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-15T22:47:24.147 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-15T22:47:28.015 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-15T22:47:29.169 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-04-15T22:47:30.911 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:47:30.920 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:47:30.920 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:47:30.920 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:47:30.920 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:47:30.920 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T22:47:30.921 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-04-15T22:47:30.922 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout:Install 28 Packages 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-04-15T22:47:30.923 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:30.924 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 53 M 2024-04-15T22:47:30.924 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:47:31.299 INFO:teuthology.orchestra.run.smithi037.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 30 MB/s | 5.2 MB 00:00 2024-04-15T22:47:31.325 INFO:teuthology.orchestra.run.smithi037.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 968 kB/s | 24 kB 00:00 2024-04-15T22:47:31.358 INFO:teuthology.orchestra.run.smithi037.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 19 MB/s | 653 kB 00:00 2024-04-15T22:47:31.392 INFO:teuthology.orchestra.run.smithi037.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-15T22:47:31.500 INFO:teuthology.orchestra.run.smithi037.stdout:(5/30): librgw2-18.2.0-0.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-04-15T22:47:31.525 INFO:teuthology.orchestra.run.smithi037.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-15T22:47:31.551 INFO:teuthology.orchestra.run.smithi037.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.6 MB/s | 119 kB 00:00 2024-04-15T22:47:31.577 INFO:teuthology.orchestra.run.smithi037.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 155 kB 00:00 2024-04-15T22:47:31.660 INFO:teuthology.orchestra.run.smithi037.stdout:(9/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 14 MB/s | 7.6 MB 00:00 2024-04-15T22:47:31.685 INFO:teuthology.orchestra.run.smithi037.stdout:(10/30): python3-rados-18.2.0-0.el9.x86_64.rpm 2.9 MB/s | 321 kB 00:00 2024-04-15T22:47:31.711 INFO:teuthology.orchestra.run.smithi037.stdout:(11/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 5.8 MB/s | 297 kB 00:00 2024-04-15T22:47:31.737 INFO:teuthology.orchestra.run.smithi037.stdout:(12/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-04-15T22:47:31.981 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:47:31.985 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-15T22:47:31.986 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T22:47:31.987 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout:Install 28 Packages 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 2 Packages 2024-04-15T22:47:31.988 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:31.989 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 53 M 2024-04-15T22:47:31.989 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:47:31.994 INFO:teuthology.orchestra.run.smithi037.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-04-15T22:47:32.020 INFO:teuthology.orchestra.run.smithi037.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 343 kB/s | 106 kB 00:00 2024-04-15T22:47:32.045 INFO:teuthology.orchestra.run.smithi037.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 560 kB/s | 159 kB 00:00 2024-04-15T22:47:32.062 INFO:teuthology.orchestra.run.smithi037.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-04-15T22:47:32.112 INFO:teuthology.orchestra.run.smithi037.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-04-15T22:47:32.137 INFO:teuthology.orchestra.run.smithi037.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-04-15T22:47:32.154 INFO:teuthology.orchestra.run.smithi037.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 447 kB/s | 41 kB 00:00 2024-04-15T22:47:32.279 INFO:teuthology.orchestra.run.smithi189.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 26 MB/s | 5.2 MB 00:00 2024-04-15T22:47:32.304 INFO:teuthology.orchestra.run.smithi189.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 958 kB/s | 24 kB 00:00 2024-04-15T22:47:32.346 INFO:teuthology.orchestra.run.smithi189.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 15 MB/s | 653 kB 00:00 2024-04-15T22:47:32.380 INFO:teuthology.orchestra.run.smithi189.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-15T22:47:32.421 INFO:teuthology.orchestra.run.smithi037.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-04-15T22:47:32.446 INFO:teuthology.orchestra.run.smithi037.stdout:(21/30): fmt-8.1.1-5.el9.x86_64.rpm 332 kB/s | 111 kB 00:00 2024-04-15T22:47:32.473 INFO:teuthology.orchestra.run.smithi189.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 19 MB/s | 7.6 MB 00:00 2024-04-15T22:47:32.480 INFO:teuthology.orchestra.run.smithi037.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-15T22:47:32.497 INFO:teuthology.orchestra.run.smithi189.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-15T22:47:32.505 INFO:teuthology.orchestra.run.smithi037.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-04-15T22:47:32.522 INFO:teuthology.orchestra.run.smithi189.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.7 MB/s | 119 kB 00:00 2024-04-15T22:47:32.530 INFO:teuthology.orchestra.run.smithi037.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-15T22:47:32.548 INFO:teuthology.orchestra.run.smithi189.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 155 kB 00:00 2024-04-15T22:47:32.573 INFO:teuthology.orchestra.run.smithi189.stdout:(9/30): python3-rados-18.2.0-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-04-15T22:47:32.607 INFO:teuthology.orchestra.run.smithi189.stdout:(10/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 8.7 MB/s | 297 kB 00:00 2024-04-15T22:47:32.631 INFO:teuthology.orchestra.run.smithi037.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-04-15T22:47:32.632 INFO:teuthology.orchestra.run.smithi189.stdout:(11/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-04-15T22:47:32.706 INFO:teuthology.orchestra.run.smithi037.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-04-15T22:47:32.731 INFO:teuthology.orchestra.run.smithi037.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 961 kB/s | 192 kB 00:00 2024-04-15T22:47:32.732 INFO:teuthology.orchestra.run.smithi189.stdout:(12/30): librgw2-18.2.0-0.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-15T22:47:32.773 INFO:teuthology.orchestra.run.smithi037.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-15T22:47:32.865 INFO:teuthology.orchestra.run.smithi037.stdout:(29/30): librbd1-18.2.0-0.el9.x86_64.rpm 23 MB/s | 3.0 MB 00:00 2024-04-15T22:47:32.923 INFO:teuthology.orchestra.run.smithi037.stdout:(30/30): librados2-18.2.0-0.el9.x86_64.rpm 17 MB/s | 3.3 MB 00:00 2024-04-15T22:47:32.930 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:47:32.930 INFO:teuthology.orchestra.run.smithi037.stdout:Total 26 MB/s | 53 MB 00:02 2024-04-15T22:47:32.931 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:47:32.932 INFO:teuthology.orchestra.run.smithi189.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-04-15T22:47:32.958 INFO:teuthology.orchestra.run.smithi189.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 710 kB/s | 159 kB 00:00 2024-04-15T22:47:32.970 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:47:32.971 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:47:32.983 INFO:teuthology.orchestra.run.smithi189.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 302 kB/s | 106 kB 00:00 2024-04-15T22:47:33.008 INFO:teuthology.orchestra.run.smithi189.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-04-15T22:47:33.059 INFO:teuthology.orchestra.run.smithi189.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-04-15T22:47:33.084 INFO:teuthology.orchestra.run.smithi189.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 554 kB/s | 41 kB 00:00 2024-04-15T22:47:33.109 INFO:teuthology.orchestra.run.smithi189.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-04-15T22:47:33.217 INFO:teuthology.orchestra.run.smithi189.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 699 kB/s | 111 kB 00:00 2024-04-15T22:47:33.268 INFO:teuthology.orchestra.run.smithi189.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-04-15T22:47:33.304 INFO:teuthology.orchestra.run.smithi189.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-04-15T22:47:33.427 INFO:teuthology.orchestra.run.smithi189.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-04-15T22:47:33.427 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:47:33.428 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:47:33.452 INFO:teuthology.orchestra.run.smithi189.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 455 kB/s | 67 kB 00:00 2024-04-15T22:47:33.553 INFO:teuthology.orchestra.run.smithi189.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-15T22:47:33.711 INFO:teuthology.orchestra.run.smithi189.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-04-15T22:47:33.786 INFO:teuthology.orchestra.run.smithi189.stdout:(27/30): librados2-18.2.0-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-04-15T22:47:33.853 INFO:teuthology.orchestra.run.smithi189.stdout:(28/30): librbd1-18.2.0-0.el9.x86_64.rpm 45 MB/s | 3.0 MB 00:00 2024-04-15T22:47:34.229 INFO:teuthology.orchestra.run.smithi189.stdout:(29/30): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-04-15T22:47:34.271 INFO:teuthology.orchestra.run.smithi189.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-04-15T22:47:34.274 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:47:34.274 INFO:teuthology.orchestra.run.smithi189.stdout:Total 23 MB/s | 53 MB 00:02 2024-04-15T22:47:34.282 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:47:34.311 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:47:34.311 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:47:34.759 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:47:34.759 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:47:34.999 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:47:35.091 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-15T22:47:35.123 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-15T22:47:35.174 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-15T22:47:35.514 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-15T22:47:35.578 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-15T22:47:35.674 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-15T22:47:35.755 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-04-15T22:47:35.787 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-15T22:47:35.850 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-15T22:47:35.878 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-15T22:47:35.931 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-04-15T22:47:35.998 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-15T22:47:36.023 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-15T22:47:36.080 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-15T22:47:36.143 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:47:36.191 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-15T22:47:36.230 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-15T22:47:36.240 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-15T22:47:36.267 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-15T22:47:36.270 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-15T22:47:36.303 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-15T22:47:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-15T22:47:36.339 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-15T22:47:36.375 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-15T22:47:36.408 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-15T22:47:36.698 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-15T22:47:36.721 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-15T22:47:36.741 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-15T22:47:36.745 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-15T22:47:36.833 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-15T22:47:36.836 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-15T22:47:36.867 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-04-15T22:47:36.912 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-15T22:47:36.912 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-04-15T22:47:36.954 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-15T22:47:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-15T22:47:37.050 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-15T22:47:37.107 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-04-15T22:47:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-15T22:47:37.203 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-15T22:47:37.262 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-15T22:47:37.318 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-15T22:47:37.348 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-15T22:47:37.367 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-15T22:47:37.402 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-15T22:47:37.403 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-15T22:47:37.444 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-15T22:47:37.471 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-04-15T22:47:37.484 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-15T22:47:37.500 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-04-15T22:47:37.511 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-15T22:47:37.555 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-15T22:47:37.588 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-15T22:47:37.891 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-15T22:47:37.918 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-15T22:47:37.993 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-15T22:47:38.022 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-04-15T22:47:38.062 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-15T22:47:38.457 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-15T22:47:38.488 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-15T22:47:38.542 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-15T22:47:38.616 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-04-15T22:47:38.649 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-04-15T22:47:39.610 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-15T22:47:39.638 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-15T22:47:40.213 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-15T22:47:40.250 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-15T22:47:40.360 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-15T22:47:40.361 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T22:47:40.361 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T22:47:40.361 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:40.401 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-15T22:47:40.785 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-15T22:47:40.820 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-15T22:47:41.385 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-15T22:47:41.423 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-15T22:47:41.498 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-15T22:47:41.499 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T22:47:41.499 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T22:47:41.499 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:41.535 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-04-15T22:47:50.442 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:51.212 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-15T22:47:51.237 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-15T22:47:51.237 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:47:51.237 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T22:47:51.238 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T22:47:51.238 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T22:47:51.238 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:51.240 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-04-15T22:47:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-04-15T22:47:51.257 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:51.324 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-15T22:47:51.324 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-15T22:47:52.024 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-15T22:47:52.050 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-15T22:47:52.051 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:47:52.051 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T22:47:52.051 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T22:47:52.051 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T22:47:52.051 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:52.051 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-15T22:47:52.128 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-15T22:47:52.128 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-04-15T22:47:52.393 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-15T22:47:52.394 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-15T22:47:52.395 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-15T22:47:52.395 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-15T22:47:52.395 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-15T22:47:52.395 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-04-15T22:47:52.395 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-15T22:47:52.395 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-04-15T22:47:53.263 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-15T22:47:53.263 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-04-15T22:47:53.263 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-04-15T22:47:53.267 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-04-15T22:47:53.267 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-04-15T22:47:53.267 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-04-15T22:47:53.268 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-04-15T22:47:53.268 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-04-15T22:47:53.268 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-04-15T22:47:53.268 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-04-15T22:47:53.268 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-15T22:47:53.268 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-04-15T22:47:53.269 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-04-15T22:47:53.269 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-04-15T22:47:53.269 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-15T22:47:53.269 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-15T22:47:53.269 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-15T22:47:53.269 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-15T22:47:53.270 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-15T22:47:53.270 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-15T22:47:53.270 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-15T22:47:53.270 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-15T22:47:53.270 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-15T22:47:53.270 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-04-15T22:47:53.271 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-15T22:47:53.272 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-04-15T22:47:53.304 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-15T22:47:53.305 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:53.305 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-04-15T22:47:53.305 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.305 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:47:53.305 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T22:47:53.305 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.306 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.306 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.306 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.306 INFO:teuthology.orchestra.run.smithi037.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-15T22:47:53.306 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T22:47:53.306 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T22:47:53.307 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T22:47:53.307 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.307 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T22:47:53.307 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T22:47:53.307 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T22:47:53.308 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T22:47:53.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:53.310 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T22:47:53.310 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T22:47:53.310 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:53.311 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:47:53.678 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-04-15T22:47:54.266 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:26 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.320 INFO:teuthology.orchestra.run.smithi189.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T22:47:54.321 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:54.322 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:47:54.432 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:47:54.433 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:47:54.433 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:47:54.433 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:47:54.433 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:47:54.433 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-04-15T22:47:54.433 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:47:54.434 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-04-15T22:47:54.435 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:47:54.435 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 40 M 2024-04-15T22:47:54.436 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 167 M 2024-04-15T22:47:54.436 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:47:54.720 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.2 MB/s | 161 kB 00:00 2024-04-15T22:47:54.722 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-test 2024-04-15T22:47:54.820 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 300 kB 00:00 2024-04-15T22:47:54.846 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 863 kB/s | 217 kB 00:00 2024-04-15T22:47:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-15T22:47:55.271 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): ceph-test-18.2.0-0.el9.x86_64.rpm 58 MB/s | 40 MB 00:00 2024-04-15T22:47:55.309 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:26 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:47:55.338 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): jq-1.6-16.el9.x86_64.rpm 380 kB/s | 187 kB 00:00 2024-04-15T22:47:55.341 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:47:55.341 INFO:teuthology.orchestra.run.smithi037.stdout:Total 45 MB/s | 40 MB 00:00 2024-04-15T22:47:55.356 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:47:55.378 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:47:55.378 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:47:55.471 INFO:teuthology.orchestra.run.smithi189.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout:Install 6 Packages 2024-04-15T22:47:55.472 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:47:55.473 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 40 M 2024-04-15T22:47:55.473 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 167 M 2024-04-15T22:47:55.473 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:47:55.669 INFO:teuthology.orchestra.run.smithi189.stdout:(1/6): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.2 MB/s | 161 kB 00:00 2024-04-15T22:47:55.727 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:47:55.727 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:47:55.752 INFO:teuthology.orchestra.run.smithi189.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-04-15T22:47:55.778 INFO:teuthology.orchestra.run.smithi189.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-04-15T22:47:55.803 INFO:teuthology.orchestra.run.smithi189.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-15T22:47:56.262 INFO:teuthology.orchestra.run.smithi189.stdout:(5/6): ceph-test-18.2.0-0.el9.x86_64.rpm 55 MB/s | 40 MB 00:00 2024-04-15T22:47:56.289 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:47:56.378 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-15T22:47:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-15T22:47:56.462 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-15T22:47:56.507 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-15T22:47:56.535 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-04-15T22:47:56.762 INFO:teuthology.orchestra.run.smithi189.stdout:(6/6): jq-1.6-16.el9.x86_64.rpm 190 kB/s | 187 kB 00:00 2024-04-15T22:47:56.765 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:47:56.765 INFO:teuthology.orchestra.run.smithi189.stdout:Total 31 MB/s | 40 MB 00:01 2024-04-15T22:47:56.780 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:47:56.801 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:47:56.802 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:47:57.141 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:47:57.141 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:47:57.699 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:47:57.788 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-15T22:47:57.835 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-15T22:47:57.875 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-15T22:47:57.920 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-15T22:47:57.948 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-04-15T22:48:00.445 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-04-15T22:48:00.520 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-04-15T22:48:01.663 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 5/6 2024-04-15T22:48:01.739 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-04-15T22:48:02.614 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-04-15T22:48:02.614 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/6 2024-04-15T22:48:02.615 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/6 2024-04-15T22:48:02.615 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-15T22:48:02.615 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-15T22:48:02.615 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-15T22:48:02.972 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-15T22:48:02.972 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:02.972 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:02.972 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:18.2.0-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-04-15T22:48:02.973 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:18.2.0-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T22:48:02.973 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T22:48:02.973 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:02.973 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:03.260 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-04-15T22:48:03.811 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 6/6 2024-04-15T22:48:03.811 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/6 2024-04-15T22:48:03.812 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/6 2024-04-15T22:48:03.812 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-15T22:48:03.812 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-15T22:48:03.812 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-15T22:48:03.846 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:35 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:04.006 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:48:04.008 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T22:48:04.009 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T22:48:04.010 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout:Install 33 Packages 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:04.011 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 30 M 2024-04-15T22:48:04.012 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 109 M 2024-04-15T22:48:04.012 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:04.191 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:18.2.0-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:18.2.0-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:04.192 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:04.400 INFO:teuthology.orchestra.run.smithi037.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 59 kB/s | 6.4 kB 00:00 2024-04-15T22:48:04.450 INFO:teuthology.orchestra.run.smithi037.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 9.0 MB/s | 1.4 MB 00:00 2024-04-15T22:48:04.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph 2024-04-15T22:48:04.501 INFO:teuthology.orchestra.run.smithi037.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 10 MB/s | 2.1 MB 00:00 2024-04-15T22:48:04.534 INFO:teuthology.orchestra.run.smithi037.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 7.1 MB/s | 240 kB 00:00 2024-04-15T22:48:04.601 INFO:teuthology.orchestra.run.smithi037.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-04-15T22:48:04.884 INFO:teuthology.orchestra.run.smithi037.stdout:(6/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 41 MB/s | 18 MB 00:00 2024-04-15T22:48:04.976 INFO:teuthology.orchestra.run.smithi037.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 90 kB/s | 32 kB 00:00 2024-04-15T22:48:05.018 INFO:teuthology.orchestra.run.smithi037.stdout:(8/33): python3-mako-1.1.4-6.el9.noarch.rpm 298 kB/s | 144 kB 00:00 2024-04-15T22:48:05.051 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:36 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:05.060 INFO:teuthology.orchestra.run.smithi037.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 272 kB/s | 47 kB 00:00 2024-04-15T22:48:05.102 INFO:teuthology.orchestra.run.smithi037.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 307 kB/s | 39 kB 00:00 2024-04-15T22:48:05.214 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:05.216 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:05.216 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:48:05.216 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T22:48:05.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T22:48:05.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout:Install 33 Packages 2024-04-15T22:48:05.219 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:05.220 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 30 M 2024-04-15T22:48:05.221 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 109 M 2024-04-15T22:48:05.221 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:05.261 INFO:teuthology.orchestra.run.smithi037.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 995 kB/s | 241 kB 00:00 2024-04-15T22:48:05.411 INFO:teuthology.orchestra.run.smithi037.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 333 kB/s | 103 kB 00:00 2024-04-15T22:48:05.453 INFO:teuthology.orchestra.run.smithi037.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.1 MB 00:00 2024-04-15T22:48:05.478 INFO:teuthology.orchestra.run.smithi037.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 572 kB/s | 124 kB 00:00 2024-04-15T22:48:05.503 INFO:teuthology.orchestra.run.smithi037.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.0 MB/s | 187 kB 00:00 2024-04-15T22:48:05.620 INFO:teuthology.orchestra.run.smithi037.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 883 kB/s | 125 kB 00:00 2024-04-15T22:48:05.754 INFO:teuthology.orchestra.run.smithi037.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-04-15T22:48:05.791 INFO:teuthology.orchestra.run.smithi189.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 63 kB/s | 6.4 kB 00:00 2024-04-15T22:48:05.841 INFO:teuthology.orchestra.run.smithi189.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 9.5 MB/s | 1.4 MB 00:00 2024-04-15T22:48:05.846 INFO:teuthology.orchestra.run.smithi037.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-04-15T22:48:05.880 INFO:teuthology.orchestra.run.smithi037.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 457 kB/s | 172 kB 00:00 2024-04-15T22:48:05.901 INFO:teuthology.orchestra.run.smithi189.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 10 MB/s | 2.1 MB 00:00 2024-04-15T22:48:05.930 INFO:teuthology.orchestra.run.smithi037.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-04-15T22:48:05.942 INFO:teuthology.orchestra.run.smithi189.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 5.6 MB/s | 240 kB 00:00 2024-04-15T22:48:05.955 INFO:teuthology.orchestra.run.smithi037.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-04-15T22:48:06.014 INFO:teuthology.orchestra.run.smithi037.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-04-15T22:48:06.018 INFO:teuthology.orchestra.run.smithi189.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-04-15T22:48:06.031 INFO:teuthology.orchestra.run.smithi037.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-04-15T22:48:06.055 INFO:teuthology.orchestra.run.smithi037.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 824 kB/s | 358 kB 00:00 2024-04-15T22:48:06.114 INFO:teuthology.orchestra.run.smithi037.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-04-15T22:48:06.139 INFO:teuthology.orchestra.run.smithi037.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 725 kB/s | 79 kB 00:00 2024-04-15T22:48:06.143 INFO:teuthology.orchestra.run.smithi189.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 720 kB/s | 144 kB 00:00 2024-04-15T22:48:06.160 INFO:teuthology.orchestra.run.smithi189.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 266 kB/s | 32 kB 00:00 2024-04-15T22:48:06.165 INFO:teuthology.orchestra.run.smithi037.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-15T22:48:06.177 INFO:teuthology.orchestra.run.smithi189.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-15T22:48:06.194 INFO:teuthology.orchestra.run.smithi189.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-15T22:48:06.198 INFO:teuthology.orchestra.run.smithi037.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 195 kB/s | 16 kB 00:00 2024-04-15T22:48:06.236 INFO:teuthology.orchestra.run.smithi189.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-04-15T22:48:06.240 INFO:teuthology.orchestra.run.smithi037.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 893 kB/s | 90 kB 00:00 2024-04-15T22:48:06.257 INFO:teuthology.orchestra.run.smithi037.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-04-15T22:48:06.332 INFO:teuthology.orchestra.run.smithi037.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 267 kB/s | 20 kB 00:00 2024-04-15T22:48:06.420 INFO:teuthology.orchestra.run.smithi189.stdout:(11/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 30 MB/s | 18 MB 00:00 2024-04-15T22:48:06.433 INFO:teuthology.orchestra.run.smithi037.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 981 kB/s | 230 kB 00:00 2024-04-15T22:48:06.461 INFO:teuthology.orchestra.run.smithi189.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 4.3 MB/s | 1.1 MB 00:00 2024-04-15T22:48:06.466 INFO:teuthology.orchestra.run.smithi037.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-04-15T22:48:06.470 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:06.470 INFO:teuthology.orchestra.run.smithi037.stdout:Total 12 MB/s | 30 MB 00:02 2024-04-15T22:48:06.485 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:06.487 INFO:teuthology.orchestra.run.smithi189.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 412 kB/s | 103 kB 00:00 2024-04-15T22:48:06.528 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:06.528 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:06.529 INFO:teuthology.orchestra.run.smithi189.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 124 kB 00:00 2024-04-15T22:48:06.562 INFO:teuthology.orchestra.run.smithi189.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-04-15T22:48:06.899 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:06.899 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:06.946 INFO:teuthology.orchestra.run.smithi189.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 104 kB/s | 43 kB 00:00 2024-04-15T22:48:07.171 INFO:teuthology.orchestra.run.smithi189.stdout:(17/33): python3-requests-2.25.1-8.el9.noarch.r 183 kB/s | 125 kB 00:00 2024-04-15T22:48:07.205 INFO:teuthology.orchestra.run.smithi189.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 267 kB/s | 172 kB 00:00 2024-04-15T22:48:07.321 INFO:teuthology.orchestra.run.smithi189.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-04-15T22:48:07.455 INFO:teuthology.orchestra.run.smithi189.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 174 kB/s | 23 kB 00:00 2024-04-15T22:48:07.489 INFO:teuthology.orchestra.run.smithi189.stdout:(21/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 34 kB/s | 11 kB 00:00 2024-04-15T22:48:07.556 INFO:teuthology.orchestra.run.smithi189.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 588 kB/s | 358 kB 00:00 2024-04-15T22:48:07.581 INFO:teuthology.orchestra.run.smithi189.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-04-15T22:48:07.623 INFO:teuthology.orchestra.run.smithi189.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-04-15T22:48:07.673 INFO:teuthology.orchestra.run.smithi189.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 396 kB/s | 46 kB 00:00 2024-04-15T22:48:07.715 INFO:teuthology.orchestra.run.smithi189.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 590 kB/s | 79 kB 00:00 2024-04-15T22:48:07.765 INFO:teuthology.orchestra.run.smithi189.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-04-15T22:48:07.857 INFO:teuthology.orchestra.run.smithi189.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 634 kB/s | 90 kB 00:00 2024-04-15T22:48:07.874 INFO:teuthology.orchestra.run.smithi189.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-04-15T22:48:07.999 INFO:teuthology.orchestra.run.smithi189.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 723 kB/s | 272 kB 00:00 2024-04-15T22:48:08.091 INFO:teuthology.orchestra.run.smithi189.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 982 kB/s | 230 kB 00:00 2024-04-15T22:48:08.125 INFO:teuthology.orchestra.run.smithi189.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-04-15T22:48:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:08.158 INFO:teuthology.orchestra.run.smithi189.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2024-04-15T22:48:08.162 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:08.162 INFO:teuthology.orchestra.run.smithi189.stdout:Total 10 MB/s | 30 MB 00:02 2024-04-15T22:48:08.177 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:08.195 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-15T22:48:08.217 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:08.218 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:08.226 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-15T22:48:08.259 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-15T22:48:08.354 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-15T22:48:08.448 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-15T22:48:08.487 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-15T22:48:08.570 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-15T22:48:08.575 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:08.576 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:08.630 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-15T22:48:08.667 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-15T22:48:08.726 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-15T22:48:08.789 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-15T22:48:08.983 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-15T22:48:09.030 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-15T22:48:09.069 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-15T22:48:09.103 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-15T22:48:09.133 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-15T22:48:09.214 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-15T22:48:09.252 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-15T22:48:09.285 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-15T22:48:09.326 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-15T22:48:09.369 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-15T22:48:09.412 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-15T22:48:09.504 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-15T22:48:09.543 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-15T22:48:09.618 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-15T22:48:09.757 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-15T22:48:09.798 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:09.861 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-15T22:48:09.890 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-15T22:48:09.893 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-15T22:48:09.926 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-15T22:48:10.017 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-15T22:48:10.054 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-04-15T22:48:10.080 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-15T22:48:10.097 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-15T22:48:10.105 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-15T22:48:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T22:48:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T22:48:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T22:48:10.106 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:10.137 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-15T22:48:10.217 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-15T22:48:10.271 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-15T22:48:10.309 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-15T22:48:10.358 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-15T22:48:10.423 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-15T22:48:10.617 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-15T22:48:10.672 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-15T22:48:10.711 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-15T22:48:10.745 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-15T22:48:10.775 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-15T22:48:10.852 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-15T22:48:10.894 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-15T22:48:10.936 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-15T22:48:10.976 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-15T22:48:11.011 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-15T22:48:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-15T22:48:11.135 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-15T22:48:11.176 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-15T22:48:11.258 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-15T22:48:11.394 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-15T22:48:11.519 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-15T22:48:11.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-04-15T22:48:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-15T22:48:11.706 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-15T22:48:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-15T22:48:11.725 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:11.725 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T22:48:11.725 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T22:48:11.725 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T22:48:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:11.732 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-15T22:48:11.732 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:11.732 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T22:48:11.732 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T22:48:11.732 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T22:48:11.732 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:12.190 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-15T22:48:12.214 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-15T22:48:12.214 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:12.214 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T22:48:12.214 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T22:48:12.214 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T22:48:12.214 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:12.433 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-15T22:48:12.456 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-15T22:48:12.456 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:12.456 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T22:48:12.457 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T22:48:12.457 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T22:48:12.457 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:12.523 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-15T22:48:13.308 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-15T22:48:13.338 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-15T22:48:13.338 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:13.338 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T22:48:13.338 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T22:48:13.338 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T22:48:13.338 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:13.791 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-15T22:48:13.816 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-15T22:48:13.816 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:13.816 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T22:48:13.816 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T22:48:13.816 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T22:48:13.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:14.026 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-15T22:48:14.050 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-15T22:48:14.050 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:14.050 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T22:48:14.050 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T22:48:14.050 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T22:48:14.050 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:14.116 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-15T22:48:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-15T22:48:14.150 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-15T22:48:14.151 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-15T22:48:14.152 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-15T22:48:14.153 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-15T22:48:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-15T22:48:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-15T22:48:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-15T22:48:14.154 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-15T22:48:15.227 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-15T22:48:15.227 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:15.227 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:15.227 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:15.227 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:15.227 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T22:48:15.228 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T22:48:15.229 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T22:48:15.230 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:15.230 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:15.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-04-15T22:48:15.824 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-15T22:48:15.824 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-04-15T22:48:15.824 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-04-15T22:48:15.824 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-15T22:48:15.825 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-15T22:48:15.826 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-15T22:48:15.826 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-15T22:48:15.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-15T22:48:15.828 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-15T22:48:15.828 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-15T22:48:16.190 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:48 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:16.294 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:16.350 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:16.351 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:16.351 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:16.417 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:16.668 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T22:48:16.669 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T22:48:16.670 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:16.671 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:16.994 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-base 2024-04-15T22:48:17.006 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:49 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:17.164 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:17.165 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:17.165 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:17.165 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 209 k 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 200 k 2024-04-15T22:48:17.166 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:17.230 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-18.2.0-0.el9.noarch.rpm 3.2 MB/s | 209 kB 00:00 2024-04-15T22:48:17.231 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:17.231 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.1 MB/s | 209 kB 00:00 2024-04-15T22:48:17.231 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:17.237 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:17.237 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:17.239 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:17.240 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:17.313 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:17.583 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:48 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:17.681 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:17.717 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:17.733 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:17.734 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:48:17.734 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:17.799 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install cephadm 2024-04-15T22:48:17.813 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:18.204 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:18.376 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:49 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:18.471 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-04-15T22:48:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:18.471 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:18.529 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:18.530 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 209 k 2024-04-15T22:48:18.531 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 200 k 2024-04-15T22:48:18.531 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:18.597 INFO:teuthology.orchestra.run.smithi189.stdout:cephadm-18.2.0-0.el9.noarch.rpm 3.1 MB/s | 209 kB 00:00 2024-04-15T22:48:18.600 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:18.600 INFO:teuthology.orchestra.run.smithi189.stdout:Total 3.0 MB/s | 209 kB 00:00 2024-04-15T22:48:18.600 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:18.606 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:18.607 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:18.607 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:18.607 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:18.650 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-04-15T22:48:18.675 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:19.134 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:19.246 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:51 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:19.404 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:19.404 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:19.404 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-04-15T22:48:19.404 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 142 k 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 399 k 2024-04-15T22:48:19.405 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:19.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 2.2 MB/s | 142 kB 00:00 2024-04-15T22:48:19.469 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:19.469 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.2 MB/s | 142 kB 00:00 2024-04-15T22:48:19.469 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:19.477 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:19.477 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:19.532 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:19.532 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:19.616 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:19.735 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:19.760 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:19.761 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:19.761 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T22:48:19.761 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:19.880 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-15T22:48:19.880 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:19.880 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:19.880 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-04-15T22:48:19.880 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:19.880 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:20.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-immutable-object-cache 2024-04-15T22:48:20.695 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:51 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:20.823 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:20.823 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:20.823 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:20.823 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:20.823 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:20.823 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:20.846 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:20.846 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 142 k 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 399 k 2024-04-15T22:48:20.847 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:20.912 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 2.2 MB/s | 142 kB 00:00 2024-04-15T22:48:20.912 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:20.912 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.1 MB/s | 142 kB 00:00 2024-04-15T22:48:20.912 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:20.919 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:20.919 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:20.972 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:20.972 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:21.064 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-04-15T22:48:21.117 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:21.178 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:21.204 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:21.204 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:48:21.204 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T22:48:21.204 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:21.654 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:53 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:21.756 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:21.811 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:21.812 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:21.812 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:21.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-04-15T22:48:22.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:22.212 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:22.213 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:22.213 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:22.213 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:22.213 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:22.462 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:54 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:22.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr 2024-04-15T22:48:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:48:22.623 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:Install 7 Packages 2024-04-15T22:48:22.624 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.0 M 2024-04-15T22:48:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 8.1 M 2024-04-15T22:48:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 212 kB/s | 23 kB 00:00 2024-04-15T22:48:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 116 kB/s | 15 kB 00:00 2024-04-15T22:48:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 10 MB/s | 1.7 MB 00:00 2024-04-15T22:48:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2024-04-15T22:48:23.061 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:54 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 119 kB/s | 31 kB 00:00 2024-04-15T22:48:23.163 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 125 kB/s | 41 kB 00:00 2024-04-15T22:48:23.215 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:23.216 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:48:23.216 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:23.277 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-dashboard 2024-04-15T22:48:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 525 kB/s | 188 kB 00:00 2024-04-15T22:48:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-04-15T22:48:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:23.750 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:23.800 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-15T22:48:23.834 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-15T22:48:23.849 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:54 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:23.881 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-15T22:48:23.912 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-15T22:48:23.941 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-04-15T22:48:24.002 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:24.003 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:24.004 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:24.005 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:24.005 INFO:teuthology.orchestra.run.smithi189.stdout:Install 7 Packages 2024-04-15T22:48:24.005 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:24.005 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 2.0 M 2024-04-15T22:48:24.005 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 8.1 M 2024-04-15T22:48:24.005 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:24.182 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-04-15T22:48:24.219 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-15T22:48:24.472 INFO:teuthology.orchestra.run.smithi189.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 215 kB/s | 23 kB 00:00 2024-04-15T22:48:24.497 INFO:teuthology.orchestra.run.smithi189.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 110 kB/s | 15 kB 00:00 2024-04-15T22:48:24.550 INFO:teuthology.orchestra.run.smithi189.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 9.3 MB/s | 1.7 MB 00:00 2024-04-15T22:48:24.598 INFO:teuthology.orchestra.run.smithi189.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 72 kB/s | 9.0 kB 00:00 2024-04-15T22:48:24.640 INFO:teuthology.orchestra.run.smithi189.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 287 kB/s | 41 kB 00:00 2024-04-15T22:48:24.682 INFO:teuthology.orchestra.run.smithi189.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 230 kB/s | 31 kB 00:00 2024-04-15T22:48:24.765 INFO:teuthology.orchestra.run.smithi189.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-04-15T22:48:24.766 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-15T22:48:24.766 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-04-15T22:48:24.767 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:24.775 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:24.775 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:24.841 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:24.842 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T22:48:25.054 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T22:48:25.055 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:25.142 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:25.217 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-15T22:48:25.251 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-15T22:48:25.263 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-15T22:48:25.315 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-15T22:48:25.354 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-15T22:48:25.391 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-04-15T22:48:25.621 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-04-15T22:48:25.653 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-15T22:48:25.854 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:57 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:26.013 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:48:26.018 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-04-15T22:48:26.019 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:26.020 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 37 M 2024-04-15T22:48:26.020 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 209 M 2024-04-15T22:48:26.020 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:26.175 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-15T22:48:26.175 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-04-15T22:48:26.175 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-04-15T22:48:26.175 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-04-15T22:48:26.176 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-15T22:48:26.176 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-15T22:48:26.176 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-15T22:48:26.376 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 254 kB/s | 32 kB 00:00 2024-04-15T22:48:26.472 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): ceph-mgr-diskprediction-local-18.2.0-0. 33 MB/s | 7.4 MB 00:00 2024-04-15T22:48:26.487 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:26.488 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:26.498 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 137 kB/s | 17 kB 00:00 2024-04-15T22:48:26.523 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 710 kB/s | 34 kB 00:00 2024-04-15T22:48:26.573 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.2 MB/s | 3.0 MB 00:00 2024-04-15T22:48:26.673 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-04-15T22:48:26.708 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-15T22:48:26.849 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-04-15T22:48:26.916 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-04-15T22:48:27.224 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-15T22:48:27.287 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:58 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:27.442 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:27.442 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T22:48:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:Install 12 Packages 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 37 M 2024-04-15T22:48:27.445 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 209 M 2024-04-15T22:48:27.445 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:27.459 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-04-15T22:48:27.542 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 313 kB/s | 195 kB 00:00 2024-04-15T22:48:27.742 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 900 kB/s | 803 kB 00:00 2024-04-15T22:48:27.744 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:27.744 INFO:teuthology.orchestra.run.smithi037.stdout:Total 21 MB/s | 37 MB 00:01 2024-04-15T22:48:27.790 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:27.816 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:27.817 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:27.972 INFO:teuthology.orchestra.run.smithi189.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-04-15T22:48:28.006 INFO:teuthology.orchestra.run.smithi189.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2024-04-15T22:48:28.098 INFO:teuthology.orchestra.run.smithi189.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.0-0. 28 MB/s | 7.4 MB 00:00 2024-04-15T22:48:28.115 INFO:teuthology.orchestra.run.smithi189.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 315 kB/s | 34 kB 00:00 2024-04-15T22:48:28.276 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:28.277 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:28.323 INFO:teuthology.orchestra.run.smithi189.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.1 MB/s | 3.0 MB 00:00 2024-04-15T22:48:28.424 INFO:teuthology.orchestra.run.smithi189.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-04-15T22:48:28.499 INFO:teuthology.orchestra.run.smithi189.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-04-15T22:48:28.532 INFO:teuthology.orchestra.run.smithi189.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.0 MB/s | 217 kB 00:00 2024-04-15T22:48:28.858 INFO:teuthology.orchestra.run.smithi189.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-04-15T22:48:28.994 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:29.073 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-15T22:48:29.105 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-15T22:48:29.149 INFO:teuthology.orchestra.run.smithi189.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 318 kB/s | 195 kB 00:00 2024-04-15T22:48:29.175 INFO:teuthology.orchestra.run.smithi189.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 775 kB/s | 245 kB 00:00 2024-04-15T22:48:29.203 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-15T22:48:29.241 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-15T22:48:29.425 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-15T22:48:29.433 INFO:teuthology.orchestra.run.smithi189.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 894 kB/s | 803 kB 00:00 2024-04-15T22:48:29.437 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:29.437 INFO:teuthology.orchestra.run.smithi189.stdout:Total 19 MB/s | 37 MB 00:01 2024-04-15T22:48:29.456 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-15T22:48:29.477 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:29.500 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:29.501 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:29.540 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-15T22:48:29.938 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:29.939 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:30.231 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-15T22:48:30.340 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-15T22:48:30.663 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:30.731 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-15T22:48:30.771 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-15T22:48:30.864 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-15T22:48:30.908 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-15T22:48:31.085 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-15T22:48:31.123 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-15T22:48:31.217 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-15T22:48:31.690 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-15T22:48:31.905 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-15T22:48:32.028 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-15T22:48:32.763 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-15T22:48:32.788 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-15T22:48:33.265 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-15T22:48:34.293 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-15T22:48:34.322 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-15T22:48:35.277 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-15T22:48:35.278 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-15T22:48:35.278 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-15T22:48:35.701 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-15T22:48:35.701 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:35.701 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:35.701 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-04-15T22:48:35.701 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:35.702 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:36.093 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-04-15T22:48:36.688 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:08 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:36.849 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:36.850 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T22:48:36.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-04-15T22:48:36.852 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:36.853 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.0 M 2024-04-15T22:48:36.853 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-04-15T22:48:36.853 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:36.892 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-15T22:48:36.893 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-15T22:48:37.121 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 432 kB/s | 47 kB 00:00 2024-04-15T22:48:37.138 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-04-15T22:48:37.155 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-04-15T22:48:37.188 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-04-15T22:48:37.213 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-15T22:48:37.239 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 211 kB 00:00 2024-04-15T22:48:37.258 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 642 kB/s | 43 kB 00:00 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T22:48:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T22:48:37.324 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T22:48:37.324 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T22:48:37.324 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:37.324 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:37.522 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-04-15T22:48:37.548 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 96 kB/s | 32 kB 00:00 2024-04-15T22:48:37.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-rook 2024-04-15T22:48:37.765 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2024-04-15T22:48:37.790 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 380 kB/s | 203 kB 00:00 2024-04-15T22:48:37.923 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 567 kB/s | 90 kB 00:00 2024-04-15T22:48:38.040 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-04-15T22:48:38.042 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:38.043 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-04-15T22:48:38.043 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:38.059 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:38.059 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:38.162 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:38.162 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:38.297 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:09 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:38.453 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T22:48:38.454 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout:Install 13 Packages 2024-04-15T22:48:38.455 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:38.456 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 2.0 M 2024-04-15T22:48:38.456 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 26 M 2024-04-15T22:48:38.456 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:38.862 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-15T22:48:38.912 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-15T22:48:38.913 INFO:teuthology.orchestra.run.smithi189.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 492 kB/s | 47 kB 00:00 2024-04-15T22:48:38.938 INFO:teuthology.orchestra.run.smithi189.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2024-04-15T22:48:38.963 INFO:teuthology.orchestra.run.smithi189.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-04-15T22:48:38.965 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-15T22:48:38.988 INFO:teuthology.orchestra.run.smithi189.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-15T22:48:39.009 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-15T22:48:39.014 INFO:teuthology.orchestra.run.smithi189.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-15T22:48:39.039 INFO:teuthology.orchestra.run.smithi189.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-04-15T22:48:39.051 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-15T22:48:39.064 INFO:teuthology.orchestra.run.smithi189.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-04-15T22:48:39.166 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-15T22:48:39.284 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-15T22:48:39.298 INFO:teuthology.orchestra.run.smithi189.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-04-15T22:48:39.323 INFO:teuthology.orchestra.run.smithi189.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-04-15T22:48:39.336 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-15T22:48:39.565 INFO:teuthology.orchestra.run.smithi189.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2024-04-15T22:48:39.590 INFO:teuthology.orchestra.run.smithi189.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 385 kB/s | 203 kB 00:00 2024-04-15T22:48:39.757 INFO:teuthology.orchestra.run.smithi189.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 468 kB/s | 90 kB 00:00 2024-04-15T22:48:39.899 INFO:teuthology.orchestra.run.smithi189.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-04-15T22:48:39.901 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:39.901 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-04-15T22:48:39.901 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:39.916 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:39.916 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:40.014 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:40.014 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-15T22:48:40.149 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-15T22:48:40.208 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-15T22:48:40.254 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-15T22:48:40.277 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-15T22:48:40.534 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:40.702 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-15T22:48:40.766 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-15T22:48:40.822 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-15T22:48:40.868 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-15T22:48:40.909 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-15T22:48:41.003 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-15T22:48:41.046 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-15T22:48:41.047 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-15T22:48:41.113 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-15T22:48:41.162 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T22:48:41.427 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:41.428 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:41.707 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-04-15T22:48:41.876 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-15T22:48:41.924 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-15T22:48:41.958 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-15T22:48:42.004 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-15T22:48:42.028 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-15T22:48:42.297 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:14 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:42.457 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:42.458 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-04-15T22:48:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:42.458 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-04-15T22:48:42.459 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:42.460 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 6.8 M 2024-04-15T22:48:42.460 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 33 M 2024-04-15T22:48:42.460 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-15T22:48:42.753 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-15T22:48:42.754 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-15T22:48:42.754 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-15T22:48:42.754 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-15T22:48:42.754 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-15T22:48:42.892 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.4 MB/s | 127 kB 00:00 2024-04-15T22:48:43.000 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-15T22:48:43.128 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-15T22:48:43.128 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:43.128 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T22:48:43.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T22:48:43.130 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:43.130 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:43.259 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-04-15T22:48:43.351 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-04-15T22:48:43.382 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-cephadm 2024-04-15T22:48:43.443 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-04-15T22:48:43.468 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 954 kB/s | 548 kB 00:00 2024-04-15T22:48:43.470 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:43.471 INFO:teuthology.orchestra.run.smithi037.stdout:Total 6.7 MB/s | 6.8 MB 00:01 2024-04-15T22:48:43.484 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:43.495 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:43.495 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:43.587 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:43.587 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:43.881 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:43.972 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:14 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:44.064 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-15T22:48:44.125 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:44.126 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:44.126 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-15T22:48:44.126 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:44.126 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:44.126 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-04-15T22:48:44.126 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout:Install 6 Packages 2024-04-15T22:48:44.127 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:44.128 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 6.8 M 2024-04-15T22:48:44.128 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 33 M 2024-04-15T22:48:44.128 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:44.128 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-15T22:48:44.686 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-15T22:48:44.774 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-15T22:48:44.832 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-15T22:48:44.858 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-15T22:48:45.401 INFO:teuthology.orchestra.run.smithi189.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.3 MB/s | 127 kB 00:00 2024-04-15T22:48:45.626 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-15T22:48:45.626 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-04-15T22:48:45.626 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-15T22:48:45.626 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-15T22:48:45.626 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-15T22:48:45.627 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-15T22:48:45.634 INFO:teuthology.orchestra.run.smithi189.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-04-15T22:48:45.660 INFO:teuthology.orchestra.run.smithi189.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 693 kB/s | 248 kB 00:00 2024-04-15T22:48:45.885 INFO:teuthology.orchestra.run.smithi189.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 231 kB/s | 58 kB 00:00 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T22:48:45.916 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:45.917 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:45.935 INFO:teuthology.orchestra.run.smithi189.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 202 kB/s | 51 kB 00:00 2024-04-15T22:48:46.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-04-15T22:48:46.729 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:18 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:46.888 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:46.888 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:46.888 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:46.888 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:46.888 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-04-15T22:48:46.889 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:46.890 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 913 k 2024-04-15T22:48:46.890 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.7 M 2024-04-15T22:48:46.890 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:47.002 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 7.3 MB/s | 835 kB 00:00 2024-04-15T22:48:47.177 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 274 kB/s | 79 kB 00:00 2024-04-15T22:48:47.179 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:47.179 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.1 MB/s | 913 kB 00:00 2024-04-15T22:48:47.179 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:47.201 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:47.201 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:47.279 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:47.279 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:47.472 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:47.604 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-15T22:48:47.671 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-15T22:48:48.345 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-15T22:48:48.345 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-04-15T22:48:48.667 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-15T22:48:48.667 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:48.668 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:48.668 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-15T22:48:48.668 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:48.668 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:48.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-04-15T22:48:49.494 INFO:teuthology.orchestra.run.smithi189.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 134 kB/s | 548 kB 00:04 2024-04-15T22:48:49.498 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:49.498 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.3 MB/s | 6.8 MB 00:05 2024-04-15T22:48:49.502 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:21 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:49.513 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:49.524 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:49.524 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:49.616 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:49.616 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:49.665 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:49.666 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:49.667 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:49.667 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:48:49.667 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:49.667 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 126 k 2024-04-15T22:48:49.667 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 456 k 2024-04-15T22:48:49.667 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:49.731 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.9 MB/s | 126 kB 00:00 2024-04-15T22:48:49.732 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:49.732 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.9 MB/s | 126 kB 00:00 2024-04-15T22:48:49.732 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:49.738 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:49.738 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:49.795 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:49.796 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:49.913 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:49.943 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:50.040 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:50.089 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-15T22:48:50.144 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-15T22:48:50.477 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:50.667 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-15T22:48:50.750 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:50.750 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:50.750 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:50.750 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:50.750 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:50.750 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:50.755 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-15T22:48:50.813 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-15T22:48:50.840 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-15T22:48:50.990 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-04-15T22:48:51.595 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:23 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:51.601 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-15T22:48:51.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-04-15T22:48:51.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-15T22:48:51.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-15T22:48:51.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-15T22:48:51.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-15T22:48:51.698 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:51.754 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:51.756 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:51.756 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:51.825 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T22:48:51.900 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T22:48:51.901 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T22:48:51.901 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T22:48:51.901 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:51.901 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:52.166 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-fuse 2024-04-15T22:48:52.417 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:24 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:52.576 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:52.577 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:52.578 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:52.578 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:48:52.578 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:52.578 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 30 k 2024-04-15T22:48:52.578 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 137 k 2024-04-15T22:48:52.578 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 515 kB/s | 30 kB 00:00 2024-04-15T22:48:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Total 504 kB/s | 30 kB 00:00 2024-04-15T22:48:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:52.641 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:52.641 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:52.671 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:52.671 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:52.743 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:23 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:52.766 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:52.880 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:52.897 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout:Install 2 Packages 2024-04-15T22:48:52.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:52.899 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 913 k 2024-04-15T22:48:52.899 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 2.7 M 2024-04-15T22:48:52.899 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:53.003 INFO:teuthology.orchestra.run.smithi189.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 7.9 MB/s | 835 kB 00:00 2024-04-15T22:48:53.062 INFO:teuthology.orchestra.run.smithi189.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 485 kB/s | 79 kB 00:00 2024-04-15T22:48:53.066 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:53.066 INFO:teuthology.orchestra.run.smithi189.stdout:Total 5.4 MB/s | 913 kB 00:00 2024-04-15T22:48:53.066 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:53.084 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:53.084 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:53.160 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:53.160 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:53.274 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:53.348 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:53.492 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-15T22:48:53.533 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:53.533 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:53.533 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:48:53.533 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:53.533 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:53.533 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:53.590 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-15T22:48:53.718 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-04-15T22:48:54.311 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:26 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:54.395 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-15T22:48:54.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-04-15T22:48:54.414 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:54.470 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:54.471 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:54.471 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:54.537 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-04-15T22:48:54.741 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-15T22:48:54.741 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:54.741 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:54.741 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-15T22:48:54.741 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:54.742 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:54.991 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados-devel 2024-04-15T22:48:55.125 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:27 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:55.229 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:55.284 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:55.285 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:55.286 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:55.355 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-04-15T22:48:55.586 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:26 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:55.740 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:55.741 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:55.741 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:55.741 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:55.742 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:55.742 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-04-15T22:48:55.742 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:55.742 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:55.742 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:55.742 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:48:55.743 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:55.743 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 126 k 2024-04-15T22:48:55.743 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 456 k 2024-04-15T22:48:55.743 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:55.807 INFO:teuthology.orchestra.run.smithi189.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.9 MB/s | 126 kB 00:00 2024-04-15T22:48:55.808 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:55.808 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.8 MB/s | 126 kB 00:00 2024-04-15T22:48:55.808 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:55.814 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:55.814 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:55.870 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:55.870 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:55.949 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:27 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:56.015 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:56.051 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:56.107 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:56.108 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:56.109 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:56.113 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:56.172 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-04-15T22:48:56.554 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:56.769 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:28 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:56.822 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:56.822 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:56.822 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:56.823 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:56.823 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:56.823 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:56.870 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:56.926 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:56.927 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:56.927 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:56.991 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-04-15T22:48:57.076 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs2 2024-04-15T22:48:57.576 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:29 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:57.669 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:28 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:57.677 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:57.734 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:57.735 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:57.735 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:57.770 INFO:teuthology.orchestra.run.smithi189.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:57.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-04-15T22:48:57.823 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:57.823 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:48:57.823 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:57.889 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs-devel 2024-04-15T22:48:58.395 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:30 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:58.465 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:29 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:48:58.497 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:48:58.552 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:58.553 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-04-15T22:48:58.554 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:48:58.618 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:48:58.618 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 30 k 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 137 k 2024-04-15T22:48:58.619 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:48:58.620 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-04-15T22:48:58.677 INFO:teuthology.orchestra.run.smithi189.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 524 kB/s | 30 kB 00:00 2024-04-15T22:48:58.678 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:58.678 INFO:teuthology.orchestra.run.smithi189.stdout:Total 512 kB/s | 30 kB 00:00 2024-04-15T22:48:58.678 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:48:58.682 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:48:58.682 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:48:58.720 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:48:58.720 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:48:58.806 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:48:58.908 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:59.212 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:31 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:48:59.268 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:59.371 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:48:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:48:59.373 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 86 k 2024-04-15T22:48:59.373 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 230 k 2024-04-15T22:48:59.373 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:48:59.432 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.4 MB/s | 86 kB 00:00 2024-04-15T22:48:59.432 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:48:59.433 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.4 MB/s | 86 kB 00:00 2024-04-15T22:48:59.433 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:48:59.440 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:48:59.440 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:48:59.498 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:48:59.498 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:48:59.539 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:59.540 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:59.540 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:48:59.540 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-04-15T22:48:59.540 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:48:59.540 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:48:59.636 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:48:59.733 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:48:59.746 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados2 2024-04-15T22:49:00.131 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:00.325 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:31 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:00.409 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:00.410 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:00.410 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:49:00.410 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-04-15T22:49:00.410 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:00.410 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:49:00.425 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:49:00.477 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:00.478 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:49:00.478 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:00.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librbd1 2024-04-15T22:49:00.680 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-04-15T22:49:01.120 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:32 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:01.222 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:49:01.272 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:33 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:49:01.274 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:01.276 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:49:01.276 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:01.343 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rados 2024-04-15T22:49:01.432 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:49:01.433 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:49:01.433 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.0 M 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 12 M 2024-04-15T22:49:01.434 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:49:01.560 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-04-15T22:49:01.560 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:49:01.560 INFO:teuthology.orchestra.run.smithi037.stdout:Total 23 MB/s | 3.0 MB 00:00 2024-04-15T22:49:01.561 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:49:01.571 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:49:01.571 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:49:01.646 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:49:01.646 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:49:01.930 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:32 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:02.031 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:49:02.077 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:49:02.084 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:02.085 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:49:02.085 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:02.127 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:02.151 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rgw 2024-04-15T22:49:02.152 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:02.152 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:49:02.152 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T22:49:02.152 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T22:49:02.152 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T22:49:02.152 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:02.736 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:33 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:02.835 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:49:02.890 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:02.891 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:49:02.891 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:02.946 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-cephfs 2024-04-15T22:49:03.273 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:03.273 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:03.273 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:49:03.273 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-04-15T22:49:03.273 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:03.273 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:49:03.525 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:34 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:03.529 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-04-15T22:49:03.625 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:49:03.678 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:03.679 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:49:03.679 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:03.747 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rbd 2024-04-15T22:49:04.125 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:36 ago on Mon 15 Apr 2024 10:47:28 PM UTC. 2024-04-15T22:49:04.284 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-04-15T22:49:04.285 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-04-15T22:49:04.286 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-04-15T22:49:04.286 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:04.286 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 169 k 2024-04-15T22:49:04.286 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 490 k 2024-04-15T22:49:04.286 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-04-15T22:49:04.336 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:35 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:04.346 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2024-04-15T22:49:04.347 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:49:04.347 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.7 MB/s | 169 kB 00:00 2024-04-15T22:49:04.347 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-04-15T22:49:04.355 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-04-15T22:49:04.355 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-04-15T22:49:04.413 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-04-15T22:49:04.413 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-04-15T22:49:04.435 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-15T22:49:04.487 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:04.488 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-15T22:49:04.489 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:04.546 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-fuse 2024-04-15T22:49:04.575 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-04-15T22:49:04.680 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:05.128 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:36 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:05.282 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:05.283 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:05.283 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:49:05.283 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 86 k 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 230 k 2024-04-15T22:49:05.284 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:49:05.355 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-04-15T22:49:05.356 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:49:05.356 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-04-15T22:49:05.356 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:49:05.363 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:49:05.363 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:49:05.418 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:49:05.418 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:49:05.456 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:05.456 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:05.456 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-04-15T22:49:05.456 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-04-15T22:49:05.456 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:05.456 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-04-15T22:49:05.561 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:49:05.662 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:05.685 DEBUG:teuthology.parallel:result is None 2024-04-15T22:49:06.063 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:06.339 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:06.339 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:06.340 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:49:06.340 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-04-15T22:49:06.340 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:06.340 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:06.548 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-mirror 2024-04-15T22:49:07.147 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:38 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:07.306 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:07.307 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:49:07.308 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:07.309 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.0 M 2024-04-15T22:49:07.309 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 12 M 2024-04-15T22:49:07.309 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:49:07.418 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 27 MB/s | 3.0 MB 00:00 2024-04-15T22:49:07.419 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:49:07.419 INFO:teuthology.orchestra.run.smithi189.stdout:Total 27 MB/s | 3.0 MB 00:00 2024-04-15T22:49:07.419 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:49:07.429 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:49:07.429 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:49:07.502 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:49:07.502 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:49:07.913 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:49:07.977 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:08.003 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:08.003 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T22:49:08.003 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T22:49:08.003 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T22:49:08.003 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T22:49:08.003 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:09.090 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:09.091 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:09.091 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:49:09.091 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-04-15T22:49:09.091 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:09.091 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:09.313 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-nbd 2024-04-15T22:49:09.904 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:40 ago on Mon 15 Apr 2024 10:47:29 PM UTC. 2024-04-15T22:49:10.059 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 169 k 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 490 k 2024-04-15T22:49:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-15T22:49:10.130 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-04-15T22:49:10.131 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:49:10.131 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.3 MB/s | 169 kB 00:00 2024-04-15T22:49:10.131 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-15T22:49:10.139 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-15T22:49:10.139 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-15T22:49:10.196 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-15T22:49:10.196 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-15T22:49:10.357 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-15T22:49:10.453 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:10.902 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:11.189 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-15T22:49:11.189 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:11.189 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-15T22:49:11.189 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-04-15T22:49:11.190 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:49:11.190 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-15T22:49:11.409 DEBUG:teuthology.parallel:result is None 2024-04-15T22:49:11.409 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-15T22:49:11.409 INFO:teuthology.packaging:ref: None 2024-04-15T22:49:11.410 INFO:teuthology.packaging:tag: v18.2.0 2024-04-15T22:49:11.410 INFO:teuthology.packaging:branch: None 2024-04-15T22:49:11.410 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:49:11.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-15T22:49:11.552 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T22:49:11.578 INFO:teuthology.orchestra.run.smithi037.stdout:18.2.0-0.el9 2024-04-15T22:49:11.579 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-04-15T22:49:11.579 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-04-15T22:49:11.580 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-15T22:49:11.581 INFO:teuthology.packaging:ref: None 2024-04-15T22:49:11.581 INFO:teuthology.packaging:tag: v18.2.0 2024-04-15T22:49:11.581 INFO:teuthology.packaging:branch: None 2024-04-15T22:49:11.581 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:49:11.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-15T22:49:11.811 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T22:49:11.832 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.0-0.el9 2024-04-15T22:49:11.832 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-04-15T22:49:11.833 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-04-15T22:49:11.835 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-15T22:49:11.835 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:49:11.835 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T22:49:11.870 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:49:11.870 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T22:49:11.901 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-15T22:49:11.901 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:49:11.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T22:49:11.946 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T22:49:12.020 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:49:12.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T22:49:12.069 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T22:49:12.138 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-15T22:49:12.138 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:49:12.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T22:49:12.171 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T22:49:12.243 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:49:12.243 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T22:49:12.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T22:49:12.341 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-15T22:49:12.342 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:49:12.342 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T22:49:12.374 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T22:49:12.444 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:49:12.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T22:49:12.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T22:49:12.543 INFO:teuthology.run_tasks:Running task print... 2024-04-15T22:49:12.554 INFO:teuthology.task.print:**** done install task... 2024-04-15T22:49:12.554 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-15T22:49:12.698 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.0', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'} 2024-04-15T22:49:12.698 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.0 2024-04-15T22:49:12.699 INFO:tasks.cephadm:Cluster fsid is 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:12.699 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-15T22:49:12.699 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-15T22:49:12.699 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi037': '172.21.15.37', 'mon.smithi189': '172.21.15.189'} 2024-04-15T22:49:12.699 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-15T22:49:12.699 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname $(hostname -s) 2024-04-15T22:49:12.734 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-04-15T22:49:12.764 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-15T22:49:12.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:49:12.984 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54895/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2114-ga9a752df', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2114.ga9a752df'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-14 16:04:37.608971', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-15T22:49:13.351 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref reef, sha1 f1f10a1c441d6255eaa3105af60f09fbca74c76c from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-15T22:49:13.353 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/reef/f1f10a1c441d6255eaa3105af60f09fbca74c76c/centos/9/x86_64/flavors/default/cephadm 2024-04-15T22:49:13.353 DEBUG:teuthology.orchestra.run.smithi037:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/f1f10a1c441d6255eaa3105af60f09fbca74c76c/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T22:49:13.964 INFO:teuthology.orchestra.run.smithi037.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 15 22:49 /home/ubuntu/cephtest/cephadm 2024-04-15T22:49:13.965 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/f1f10a1c441d6255eaa3105af60f09fbca74c76c/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T22:49:14.705 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 Apr 15 22:49 /home/ubuntu/cephtest/cephadm 2024-04-15T22:49:14.706 DEBUG:teuthology.orchestra.run.smithi037:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T22:49:14.725 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T22:49:14.755 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.0 on all hosts... 2024-04-15T22:49:14.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-04-15T22:49:14.768 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-04-15T22:49:15.572 INFO:teuthology.orchestra.run.smithi037.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-15T22:49:15.706 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-15T22:49:36.385 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-15T22:49:36.385 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-15T22:49:36.386 INFO:teuthology.orchestra.run.smithi037.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-04-15T22:49:36.386 INFO:teuthology.orchestra.run.smithi037.stdout: "repo_digests": [ 2024-04-15T22:49:36.386 INFO:teuthology.orchestra.run.smithi037.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-04-15T22:49:36.386 INFO:teuthology.orchestra.run.smithi037.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-04-15T22:49:36.386 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-15T22:49:36.386 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-15T22:49:38.012 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-15T22:49:38.031 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph 2024-04-15T22:49:38.068 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-04-15T22:49:38.099 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /etc/ceph 2024-04-15T22:49:38.140 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-04-15T22:49:38.168 INFO:tasks.cephadm:Writing seed config... 2024-04-15T22:49:38.169 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-15T22:49:38.169 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-15T22:49:38.169 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-15T22:49:38.169 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-15T22:49:38.170 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-15T22:49:38.171 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-15T22:49:38.172 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-15T22:49:38.173 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:49:38.173 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-15T22:49:38.196 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 60ff499a-fb7a-11ee-bc8f-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-15T22:49:38.197 DEBUG:teuthology.orchestra.run.smithi037:mon.smithi037> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi037.service 2024-04-15T22:49:38.239 INFO:tasks.cephadm:Bootstrapping... 2024-04-15T22:49:38.240 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 -v bootstrap --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.37 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:49:38.411 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:49:38.412 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.0', '-v', 'bootstrap', '--fsid', '60ff499a-fb7a-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.37', '--skip-admin-label'] 2024-04-15T22:49:38.437 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-15T22:49:38.437 INFO:teuthology.orchestra.run.smithi037.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-15T22:49:38.438 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying podman|docker is present... 2024-04-15T22:49:38.461 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-15T22:49:38.461 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying lvm2 is present... 2024-04-15T22:49:38.461 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying time synchronization is in place... 2024-04-15T22:49:38.469 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-15T22:49:38.469 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T22:49:38.475 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-15T22:49:38.475 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-04-15T22:49:38.481 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout enabled 2024-04-15T22:49:38.488 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout active 2024-04-15T22:49:38.488 INFO:teuthology.orchestra.run.smithi037.stdout:Unit chronyd.service is enabled and running 2024-04-15T22:49:38.488 INFO:teuthology.orchestra.run.smithi037.stdout:Repeating the final host check... 2024-04-15T22:49:38.511 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-15T22:49:38.511 INFO:teuthology.orchestra.run.smithi037.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-15T22:49:38.511 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl is present 2024-04-15T22:49:38.511 INFO:teuthology.orchestra.run.smithi037.stdout:lvcreate is present 2024-04-15T22:49:38.518 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-15T22:49:38.518 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T22:49:38.524 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-15T22:49:38.524 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-04-15T22:49:38.530 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout enabled 2024-04-15T22:49:38.538 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout active 2024-04-15T22:49:38.538 INFO:teuthology.orchestra.run.smithi037.stdout:Unit chronyd.service is enabled and running 2024-04-15T22:49:38.538 INFO:teuthology.orchestra.run.smithi037.stdout:Host looks OK 2024-04-15T22:49:38.539 INFO:teuthology.orchestra.run.smithi037.stdout:Cluster fsid: 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:38.539 INFO:teuthology.orchestra.run.smithi037.stdout:Acquiring lock 139671852183904 on /run/cephadm/60ff499a-fb7a-11ee-bc8f-c7b262605968.lock 2024-04-15T22:49:38.539 INFO:teuthology.orchestra.run.smithi037.stdout:Lock 139671852183904 acquired on /run/cephadm/60ff499a-fb7a-11ee-bc8f-c7b262605968.lock 2024-04-15T22:49:38.539 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying IP 172.21.15.37 port 3300 ... 2024-04-15T22:49:38.539 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying IP 172.21.15.37 port 6789 ... 2024-04-15T22:49:38.539 INFO:teuthology.orchestra.run.smithi037.stdout:Base mon IP(s) is [172.21.15.37:3300, 172.21.15.37:6789], mon addrv is [v2:172.21.15.37:3300,v1:172.21.15.37:6789] 2024-04-15T22:49:38.541 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.37 metric 100 2024-04-15T22:49:38.541 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.37 metric 100 2024-04-15T22:49:38.543 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-15T22:49:38.544 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-15T22:49:38.544 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-15T22:49:38.546 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-15T22:49:38.546 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-15T22:49:38.546 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-15T22:49:38.546 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-15T22:49:38.546 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7794/64 scope link noprefixroute 2024-04-15T22:49:38.546 INFO:teuthology.orchestra.run.smithi037.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-15T22:49:38.547 INFO:teuthology.orchestra.run.smithi037.stdout:Mon IP `172.21.15.37` is in CIDR network `172.21.0.0/20` 2024-04-15T22:49:38.547 INFO:teuthology.orchestra.run.smithi037.stdout:Mon IP `172.21.15.37` is in CIDR network `172.21.0.0/20` 2024-04-15T22:49:38.547 INFO:teuthology.orchestra.run.smithi037.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-15T22:49:38.547 INFO:teuthology.orchestra.run.smithi037.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-15T22:49:38.548 INFO:teuthology.orchestra.run.smithi037.stdout:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stdout dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.0... 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Getting image source signatures 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Copying blob sha256:3bd20aeff60302f668275dc2005d10679ae56492967a3a5a54fd3dde85333aec 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Copying config sha256:dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-04-15T22:49:39.596 INFO:teuthology.orchestra.run.smithi037.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-15T22:49:40.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph: stdout ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-04-15T22:49:40.609 INFO:teuthology.orchestra.run.smithi037.stdout:Ceph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-04-15T22:49:40.609 INFO:teuthology.orchestra.run.smithi037.stdout:Extracting ceph user uid/gid from container image... 2024-04-15T22:49:41.572 INFO:teuthology.orchestra.run.smithi037.stdout:stat: stdout 167 167 2024-04-15T22:49:41.573 INFO:teuthology.orchestra.run.smithi037.stdout:Creating initial keys... 2024-04-15T22:49:42.445 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph-authtool: stdout AQAFrx1m1zJFOBAA0cNimK/gSRvfltQGVLwUew== 2024-04-15T22:49:43.341 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph-authtool: stdout AQAGrx1mGEciLBAAtchdAkb/SByFFY+LiLCKCg== 2024-04-15T22:49:44.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph-authtool: stdout AQAHrx1mswFyKhAAr/Trd7nT1G/zX3CBjnSIPg== 2024-04-15T22:49:44.499 INFO:teuthology.orchestra.run.smithi037.stdout:Creating initial monmap... 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool for smithi037 [v2:172.21.15.37:3300,v1:172.21.15.37:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = pacific 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: set fsid to 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T22:49:45.320 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:49:45.321 INFO:teuthology.orchestra.run.smithi037.stdout:Creating mon... 2024-04-15T22:49:46.309 INFO:teuthology.orchestra.run.smithi037.stdout:create mon.smithi037 on 2024-04-15T22:49:46.524 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-15T22:49:46.672 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-15T22:49:46.861 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968.target. 2024-04-15T22:49:46.861 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968.target. 2024-04-15T22:49:47.077 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi037 2024-04-15T22:49:47.077 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to reset failed state of unit ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi037.service: Unit ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi037.service not loaded. 2024-04-15T22:49:47.249 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968.target.wants/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi037.service → /etc/systemd/system/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@.service. 2024-04-15T22:49:47.772 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T22:49:47.772 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-04-15T22:49:47.779 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T22:49:47.779 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-04-15T22:49:47.779 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-04-15T22:49:47.779 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to enable service . firewalld.service is not available 2024-04-15T22:49:47.779 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mon to start... 2024-04-15T22:49:47.779 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mon... 2024-04-15T22:49:48.812 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:48 smithi037 ceph-mon[28821]: from='client.? 172.21.15.37:0/2026412170' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-15T22:49:48.822 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout cluster: 2024-04-15T22:49:48.822 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout id: 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:48.822 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-15T22:49:48.822 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:48.822 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout services: 2024-04-15T22:49:48.822 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi037 (age 0.603009s) 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout data: 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout pgs: 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.211+0000 7fd143d6f700 1 Processor -- start 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.211+0000 7fd143d6f700 1 -- start start 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.211+0000 7fd143d6f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.211+0000 7fd143d6f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c106460 con 0x7fd13c105b10 2024-04-15T22:49:48.823 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.211+0000 7fd141b0b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd141b0b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47766/0 (socket says 172.21.15.37:47766) 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd141b0b700 1 -- 172.21.15.37:0/2426375737 learned_addr learned my addr 172.21.15.37:0/2426375737 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd141b0b700 1 -- 172.21.15.37:0/2426375737 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c1065a0 con 0x7fd13c105b10 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd141b0b700 1 --2- 172.21.15.37:0/2426375737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd1380098a0 tx=0x7fd138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42747cc85187b493 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd140b09700 1 -- 172.21.15.37:0/2426375737 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd138004180 con 0x7fd13c105b10 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd140b09700 1 -- 172.21.15.37:0/2426375737 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1380042e0 con 0x7fd13c105b10 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.212+0000 7fd140b09700 1 -- 172.21.15.37:0/2426375737 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1380036a0 con 0x7fd13c105b10 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 -- 172.21.15.37:0/2426375737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 msgr2=0x7fd13c105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 --2- 172.21.15.37:0/2426375737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd1380098a0 tx=0x7fd138009bb0 comp rx=0 tx=0).stop 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 -- 172.21.15.37:0/2426375737 shutdown_connections 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 --2- 172.21.15.37:0/2426375737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c105f20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:48.824 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 -- 172.21.15.37:0/2426375737 >> 172.21.15.37:0/2426375737 conn(0x7fd13c101180 msgr2=0x7fd13c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 -- 172.21.15.37:0/2426375737 shutdown_connections 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 -- 172.21.15.37:0/2426375737 wait complete. 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.213+0000 7fd143d6f700 1 Processor -- start 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd143d6f700 1 -- start start 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd143d6f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd143d6f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c199550 con 0x7fd13c105b10 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd141b0b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd141b0b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47768/0 (socket says 172.21.15.37:47768) 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd141b0b700 1 -- 172.21.15.37:0/2026412170 learned_addr learned my addr 172.21.15.37:0/2026412170 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd141b0b700 1 -- 172.21.15.37:0/2026412170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd138009550 con 0x7fd13c105b10 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd141b0b700 1 --2- 172.21.15.37:0/2026412170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c199010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd13c106840 tx=0x7fd138004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd138003ce0 con 0x7fd13c105b10 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138020070 con 0x7fd13c105b10 2024-04-15T22:49:48.825 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c199750 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.214+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd138011630 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.215+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c199b70 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.215+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd138022030 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.215+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd138025d00 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.215+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c1927b0 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.218+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fd138025480 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.292+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd13c02cc30 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.293+0000 7fd132ffd700 1 -- 172.21.15.37:0/2026412170 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fd138011c20 con 0x7fd13c105b10 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.293+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 msgr2=0x7fd13c199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.293+0000 7fd143d6f700 1 --2- 172.21.15.37:0/2026412170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c199010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd13c106840 tx=0x7fd138004750 comp rx=0 tx=0).stop 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.293+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 shutdown_connections 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.293+0000 7fd143d6f700 1 --2- 172.21.15.37:0/2026412170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd13c105b10 0x7fd13c199010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:48.826 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.293+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 >> 172.21.15.37:0/2026412170 conn(0x7fd13c101180 msgr2=0x7fd13c18d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:48.827 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.294+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 shutdown_connections 2024-04-15T22:49:48.827 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:48.294+0000 7fd143d6f700 1 -- 172.21.15.37:0/2026412170 wait complete. 2024-04-15T22:49:48.827 INFO:teuthology.orchestra.run.smithi037.stdout:mon is available 2024-04-15T22:49:48.827 INFO:teuthology.orchestra.run.smithi037.stdout:Assimilating anything we can from ceph.conf... 2024-04-15T22:49:49.884 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:49 smithi037 ceph-mon[28821]: from='client.? 172.21.15.37:0/135311005' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T22:49:49.885 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:49 smithi037 ceph-mon[28821]: from='client.? 172.21.15.37:0/135311005' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [global] 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout fsid = 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.37:3300,v1:172.21.15.37:6789] 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-15T22:49:49.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [osd] 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.235+0000 7f36d4b02700 1 Processor -- start 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36d4b02700 1 -- start start 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36d4b02700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d007a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36d4b02700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d007a870 con 0x7f36d007be30 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36ce59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d007a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36ce59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d007a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47776/0 (socket says 172.21.15.37:47776) 2024-04-15T22:49:49.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36ce59c700 1 -- 172.21.15.37:0/596170828 learned_addr learned my addr 172.21.15.37:0/596170828 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36ce59c700 1 -- 172.21.15.37:0/596170828 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d007a9b0 con 0x7f36d007be30 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.236+0000 7f36ce59c700 1 --2- 172.21.15.37:0/596170828 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d007a330 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f36c00098a0 tx=0x7f36c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4417c2ebd79e2704 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36cd59a700 1 -- 172.21.15.37:0/596170828 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c0004180 con 0x7f36d007be30 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36cd59a700 1 -- 172.21.15.37:0/596170828 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c00042e0 con 0x7f36d007be30 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36cd59a700 1 -- 172.21.15.37:0/596170828 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c00036a0 con 0x7f36d007be30 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 -- 172.21.15.37:0/596170828 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 msgr2=0x7f36d007a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 --2- 172.21.15.37:0/596170828 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d007a330 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f36c00098a0 tx=0x7f36c0009bb0 comp rx=0 tx=0).stop 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 -- 172.21.15.37:0/596170828 shutdown_connections 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 --2- 172.21.15.37:0/596170828 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d007a330 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:49.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 -- 172.21.15.37:0/596170828 >> 172.21.15.37:0/596170828 conn(0x7f36d0101180 msgr2=0x7f36d0103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 -- 172.21.15.37:0/596170828 shutdown_connections 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.237+0000 7f36d4b02700 1 -- 172.21.15.37:0/596170828 wait complete. 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.238+0000 7f36d4b02700 1 Processor -- start 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.238+0000 7f36d4b02700 1 -- start start 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.238+0000 7f36d4b02700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.238+0000 7f36d4b02700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d019b770 con 0x7f36d007be30 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36ce59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36ce59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47786/0 (socket says 172.21.15.37:47786) 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36ce59c700 1 -- 172.21.15.37:0/135311005 learned_addr learned my addr 172.21.15.37:0/135311005 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36ce59c700 1 -- 172.21.15.37:0/135311005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0009550 con 0x7f36d007be30 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36ce59c700 1 --2- 172.21.15.37:0/135311005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d019b230 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f36c0004000 tx=0x7f36c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:49.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c0003ce0 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0020070 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d019b970 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c0011630 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d019bd90 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f36c0022030 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.239+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f36c0025d00 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.240+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d00623f0 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.242+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f36d00623f0 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.313+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f36d002ce40 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.317+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36c00254e0 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.317+0000 7f36c77fe700 1 -- 172.21.15.37:0/135311005 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f36c001c450 con 0x7f36d007be30 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 msgr2=0x7f36d019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:49.898 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 --2- 172.21.15.37:0/135311005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d019b230 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f36c0004000 tx=0x7f36c0004750 comp rx=0 tx=0).stop 2024-04-15T22:49:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 shutdown_connections 2024-04-15T22:49:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 --2- 172.21.15.37:0/135311005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36d007be30 0x7f36d019b230 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 >> 172.21.15.37:0/135311005 conn(0x7f36d0101180 msgr2=0x7f36d0102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 shutdown_connections 2024-04-15T22:49:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:49.318+0000 7f36d4b02700 1 -- 172.21.15.37:0/135311005 wait complete. 2024-04-15T22:49:49.899 INFO:teuthology.orchestra.run.smithi037.stdout:Generating new minimal ceph.conf... 2024-04-15T22:49:50.849 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:50 smithi037 ceph-mon[28821]: from='client.? 172.21.15.37:0/435123884' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:49:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.299+0000 7f7426b90700 1 Processor -- start 2024-04-15T22:49:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.299+0000 7f7426b90700 1 -- start start 2024-04-15T22:49:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f7426b90700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420106410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f7426b90700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74201069e0 con 0x7f7420106000 2024-04-15T22:49:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f742492c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f742492c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420106410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58132/0 (socket says 172.21.15.37:58132) 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f742492c700 1 -- 172.21.15.37:0/2587984960 learned_addr learned my addr 172.21.15.37:0/2587984960 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f742492c700 1 -- 172.21.15.37:0/2587984960 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420106b20 con 0x7f7420106000 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f742492c700 1 --2- 172.21.15.37:0/2587984960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420106410 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f74140098a0 tx=0x7f7414009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8cdb4d52f2ce2a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f741f7fe700 1 -- 172.21.15.37:0/2587984960 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414004180 con 0x7f7420106000 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f741f7fe700 1 -- 172.21.15.37:0/2587984960 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f74140042e0 con 0x7f7420106000 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.300+0000 7f741f7fe700 1 -- 172.21.15.37:0/2587984960 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414011a50 con 0x7f7420106000 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 -- 172.21.15.37:0/2587984960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 msgr2=0x7f7420106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 --2- 172.21.15.37:0/2587984960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420106410 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f74140098a0 tx=0x7f7414009bb0 comp rx=0 tx=0).stop 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 -- 172.21.15.37:0/2587984960 shutdown_connections 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 --2- 172.21.15.37:0/2587984960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420106410 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 -- 172.21.15.37:0/2587984960 >> 172.21.15.37:0/2587984960 conn(0x7f74200ff7e0 msgr2=0x7f7420101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 -- 172.21.15.37:0/2587984960 shutdown_connections 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.301+0000 7f7426b90700 1 -- 172.21.15.37:0/2587984960 wait complete. 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f7426b90700 1 Processor -- start 2024-04-15T22:49:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f7426b90700 1 -- start start 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f7426b90700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f7426b90700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7420199da0 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f742492c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f742492c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58144/0 (socket says 172.21.15.37:58144) 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f742492c700 1 -- 172.21.15.37:0/435123884 learned_addr learned my addr 172.21.15.37:0/435123884 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f742492c700 1 -- 172.21.15.37:0/435123884 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414009550 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.302+0000 7f742492c700 1 --2- 172.21.15.37:0/435123884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420199860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f74140119e0 tx=0x7f7414011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414003b90 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f741401a8a0 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7420199fa0 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414019930 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742019a3c0 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f741402d440 con 0x7f7420106000 2024-04-15T22:49:50.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.303+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f74140248e0 con 0x7f7420106000 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.304+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74201058c0 con 0x7f7420106000 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.306+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f741401aa10 con 0x7f7420106000 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.377+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f742019d450 con 0x7f7420106000 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.377+0000 7f741dffb700 1 -- 172.21.15.37:0/435123884 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f741401abb0 con 0x7f7420106000 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 msgr2=0x7f7420199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 --2- 172.21.15.37:0/435123884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420199860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f74140119e0 tx=0x7f7414011ac0 comp rx=0 tx=0).stop 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 shutdown_connections 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 --2- 172.21.15.37:0/435123884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7420106000 0x7f7420199860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 >> 172.21.15.37:0/435123884 conn(0x7f74200ff7e0 msgr2=0x7f7420104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 shutdown_connections 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:50.378+0000 7f7426b90700 1 -- 172.21.15.37:0/435123884 wait complete. 2024-04-15T22:49:50.859 INFO:teuthology.orchestra.run.smithi037.stdout:Restarting the monitor... 2024-04-15T22:49:51.103 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:50 smithi037 systemd[1]: Stopping Ceph mon.smithi037 for 60ff499a-fb7a-11ee-bc8f-c7b262605968... 2024-04-15T22:49:51.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037[28817]: 2024-04-15T22:49:51.105+0000 7f3d12969700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi037 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:49:51.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037[28817]: 2024-04-15T22:49:51.105+0000 7f3d12969700 -1 mon.smithi037@0(leader) e1 *** Got Signal Terminated *** 2024-04-15T22:49:51.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 podman[29066]: 2024-04-15 22:49:51.218264609 +0000 UTC m=+0.198080769 container died e954d08be5ea9512fdc2c850f027bb71b42cc19b1209007e8bdbd03a5e519ac2 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=HEAD) 2024-04-15T22:49:51.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 podman[29066]: 2024-04-15 22:49:51.521446429 +0000 UTC m=+0.501262591 container cleanup e954d08be5ea9512fdc2c850f027bb71b42cc19b1209007e8bdbd03a5e519ac2 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS) 2024-04-15T22:49:51.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 bash[29066]: ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037 2024-04-15T22:49:51.821 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 podman[29080]: 2024-04-15 22:49:51.722583375 +0000 UTC m=+0.504583889 container remove e954d08be5ea9512fdc2c850f027bb71b42cc19b1209007e8bdbd03a5e519ac2 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-15T22:49:52.136 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 systemd[1]: ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi037.service: Deactivated successfully. 2024-04-15T22:49:52.137 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 systemd[1]: Stopped Ceph mon.smithi037 for 60ff499a-fb7a-11ee-bc8f-c7b262605968. 2024-04-15T22:49:52.137 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:51 smithi037 systemd[1]: Starting Ceph mon.smithi037 for 60ff499a-fb7a-11ee-bc8f-c7b262605968... 2024-04-15T22:49:52.137 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 podman[29172]: 2024-04-15 22:49:52.048300739 +0000 UTC m=+0.097203781 container create f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c (image=quay.io/ceph/ceph:v18.2.0, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-15T22:49:52.137 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 podman[29172]: 2024-04-15 22:49:51.968862825 +0000 UTC m=+0.017765871 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-04-15T22:49:52.327 INFO:teuthology.orchestra.run.smithi037.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 podman[29172]: 2024-04-15 22:49:52.242417882 +0000 UTC m=+0.291320927 container init f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c (image=quay.io/ceph/ceph:v18.2.0, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2) 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 podman[29172]: 2024-04-15 22:49:52.245463173 +0000 UTC m=+0.294366218 container start f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c (image=quay.io/ceph/ceph:v18.2.0, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mon, pid 2 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: pidfile_write: ignore empty --pid-file 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: load: jerasure load: lrc 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: RocksDB version: 7.9.2 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Git sha 0 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Compile date 2023-08-03 19:21:13 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: DB SUMMARY 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: DB Session ID: ZBWCKR0TJ50S9DPOO9TV 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: CURRENT file: CURRENT 2024-04-15T22:49:52.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi037/store.db dir, Total Num: 1, files: 000008.sst 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi037/store.db: 000009.log size: 98252 ; 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.error_if_exists: 0 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.create_if_missing: 0 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.paranoid_checks: 1 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.env: 0x559ef8dc0720 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.fs: PosixFileSystem 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.info_log: 0x559efad7d2c0 2024-04-15T22:49:52.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T22:49:52.551 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.statistics: (nil) 2024-04-15T22:49:52.551 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.use_fsync: 0 2024-04-15T22:49:52.552 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_log_file_size: 0 2024-04-15T22:49:52.552 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T22:49:52.552 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T22:49:52.552 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T22:49:52.552 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T22:49:52.552 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.allow_fallocate: 1 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.use_direct_reads: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.db_log_dir: 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.wal_dir: 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T22:49:52.553 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.write_buffer_manager: 0x559ef9fee3c0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.unordered_write: 0 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T22:49:52.554 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.row_cache: None 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.wal_filter: None 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.two_write_queues: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.wal_compression: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.atomic_flush: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.log_readahead_size: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T22:49:52.555 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_background_jobs: 2 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_background_compactions: -1 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_subcompactions: 1 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T22:49:52.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_open_files: -1 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_background_flushes: -1 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Compression algorithms supported: 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kZSTD supported: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kXpressCompression supported: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kZlibCompression supported: 1 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kSnappyCompression supported: 1 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kLZ4Compression supported: 1 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: kBZip2Compression supported: 0 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-15T22:49:52.557 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000010 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.merge_operator: 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_filter: None 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559efad7d3c0) 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: cache_index_and_filter_blocks: 1 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: pin_top_level_index_and_filter: 1 2024-04-15T22:49:52.558 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: index_type: 0 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: data_block_index_type: 0 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: index_shortening: 1 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: checksum: 4 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: no_block_cache: 0 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_cache: 0x559efa08f1f0 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_cache_name: BinnedLRUCache 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_cache_options: 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: capacity : 536870912 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: num_shard_bits : 4 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: strict_capacity_limit : 0 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: high_pri_pool_ratio: 0.000 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_cache_compressed: (nil) 2024-04-15T22:49:52.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: persistent_cache: (nil) 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_size: 4096 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_size_deviation: 10 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_restart_interval: 16 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: index_block_restart_interval: 1 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: metadata_block_size: 4096 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: partition_filters: 0 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: use_delta_encoding: 1 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: filter_policy: bloomfilter 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: whole_key_filtering: 1 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: verify_compression: 0 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: read_amp_bytes_per_bit: 0 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: format_version: 5 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: enable_index_compression: 1 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: block_align: 0 2024-04-15T22:49:52.560 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: max_auto_readahead_size: 262144 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: prepopulate_block_cache: 0 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: initial_auto_readahead_size: 8192 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: num_file_reads_for_auto_readahead: 2 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression: NoCompression 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.num_levels: 7 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T22:49:52.561 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-15T22:49:52.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T22:49:52.563 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.arena_block_size: 1048576 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T22:49:52.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.table_properties_collectors: 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.inplace_update_support: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.bloom_locality: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.max_successive_merges: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T22:49:52.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.ttl: 2592000 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.enable_blob_files: false 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.min_blob_size: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.blob_file_starting_level: 0 2024-04-15T22:49:52.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ad4b5e82-1967-4c54-8725-b859c9f743b5 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713221392274088, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713221392275056, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93503, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91525, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84134, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713221392, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ad4b5e82-1967-4c54-8725-b859c9f743b5", "db_session_id": "ZBWCKR0TJ50S9DPOO9TV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713221392275139, "job": 1, "event": "recovery_finished"} 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi037/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559efa12c000 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: DB pointer 0x559efa118000 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-15T22:49:52.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: ** DB Stats ** 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: ** Compaction Stats [default] ** 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: L0 2/0 93.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Sum 2/0 93.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T22:49:52.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: ** Compaction Stats [default] ** 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:49:52.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Cumulative compaction: 0.00 GB write, 29.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Interval compaction: 0.00 GB write, 29.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Block cache BinnedLRUCache@0x559efa08f1f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: starting mon.smithi037 rank 0 at public addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] at bind addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi037 fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???) e1 preinit fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).mds e1 new map 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).mds e1 print_map 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: e1 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: legacy client fscid: -1 2024-04-15T22:49:52.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout: No filesystems configured 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expand map: {default=false} 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta from 'false' to 'false' 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expanded map: {default=false} 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expand map: {default=info} 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta from 'info' to 'info' 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expanded map: {default=info} 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expand map: {default=daemon} 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-15T22:49:52.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expanded map: {default=daemon} 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expand map: {default=debug} 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta from 'debug' to 'debug' 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: expand_channel_meta expanded map: {default=debug} 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 bash[29172]: f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 systemd[1]: Started Ceph mon.smithi037 for 60ff499a-fb7a-11ee-bc8f-c7b262605968. 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mon.smithi037 is new leader, mons smithi037 in quorum (ranks 0) 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: monmap e1: 1 mons at {smithi037=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0]} removed_ranks: {} 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: fsmap 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T22:49:52.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:52 smithi037 ceph-mon[29191]: mgrmap e1: no daemons active 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f601deed700 1 Processor -- start 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f601deed700 1 -- start start 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f601deed700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f601deed700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6018104750 con 0x7f6018103e00 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f60177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f60177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58148/0 (socket says 172.21.15.37:58148) 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.759+0000 7f60177fe700 1 -- 172.21.15.37:0/385261593 learned_addr learned my addr 172.21.15.37:0/385261593 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:53.420 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.760+0000 7f60177fe700 1 -- 172.21.15.37:0/385261593 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6018104890 con 0x7f6018103e00 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.760+0000 7f60177fe700 1 --2- 172.21.15.37:0/385261593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018104210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f60080098a0 tx=0x7f6008009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed35e18e2402a280 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.760+0000 7f60167fc700 1 -- 172.21.15.37:0/385261593 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6008004180 con 0x7f6018103e00 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.760+0000 7f60167fc700 1 -- 172.21.15.37:0/385261593 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f60080042e0 con 0x7f6018103e00 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.760+0000 7f60167fc700 1 -- 172.21.15.37:0/385261593 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6008011a50 con 0x7f6018103e00 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.760+0000 7f601deed700 1 -- 172.21.15.37:0/385261593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 msgr2=0x7f6018104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 --2- 172.21.15.37:0/385261593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018104210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f60080098a0 tx=0x7f6008009bb0 comp rx=0 tx=0).stop 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 -- 172.21.15.37:0/385261593 shutdown_connections 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 --2- 172.21.15.37:0/385261593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018104210 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 -- 172.21.15.37:0/385261593 >> 172.21.15.37:0/385261593 conn(0x7f60180ff7e0 msgr2=0x7f6018101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 -- 172.21.15.37:0/385261593 shutdown_connections 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 -- 172.21.15.37:0/385261593 wait complete. 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 Processor -- start 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.761+0000 7f601deed700 1 -- start start 2024-04-15T22:49:53.421 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f601deed700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:53.422 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f601deed700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6018106cd0 con 0x7f6018103e00 2024-04-15T22:49:53.422 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f60177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:53.422 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f60177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58158/0 (socket says 172.21.15.37:58158) 2024-04-15T22:49:53.422 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f60177fe700 1 -- 172.21.15.37:0/1233739722 learned_addr learned my addr 172.21.15.37:0/1233739722 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:53.422 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f60177fe700 1 -- 172.21.15.37:0/1233739722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6008009550 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f60177fe700 1 --2- 172.21.15.37:0/1233739722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018106790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6008006af0 tx=0x7f6008011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6008003b90 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f600801a8a0 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6008019880 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6018104de0 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.762+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6018105200 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.763+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f600802d440 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.763+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6008024860 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.763+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60181079c0 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.766+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f600801aa10 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.837+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6018105860 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.842+0000 7f6014ff9700 1 -- 172.21.15.37:0/1233739722 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f600801abb0 con 0x7f6018103e00 2024-04-15T22:49:53.423 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.842+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 msgr2=0x7f6018106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.842+0000 7f601deed700 1 --2- 172.21.15.37:0/1233739722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018106790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6008006af0 tx=0x7f6008011a90 comp rx=0 tx=0).stop 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.842+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 shutdown_connections 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.842+0000 7f601deed700 1 --2- 172.21.15.37:0/1233739722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6018103e00 0x7f6018106790 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.842+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 >> 172.21.15.37:0/1233739722 conn(0x7f60180ff7e0 msgr2=0x7f6018100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.843+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 shutdown_connections 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:52.843+0000 7f601deed700 1 -- 172.21.15.37:0/1233739722 wait complete. 2024-04-15T22:49:53.424 INFO:teuthology.orchestra.run.smithi037.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-15T22:49:53.446 INFO:teuthology.orchestra.run.smithi037.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:49:53.446 INFO:teuthology.orchestra.run.smithi037.stdout:Creating mgr... 2024-04-15T22:49:53.447 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-15T22:49:53.447 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-15T22:49:53.447 INFO:teuthology.orchestra.run.smithi037.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-15T22:49:53.686 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mgr.smithi037.gtwkck 2024-04-15T22:49:53.686 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Failed to reset failed state of unit ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mgr.smithi037.gtwkck.service: Unit ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mgr.smithi037.gtwkck.service not loaded. 2024-04-15T22:49:53.828 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968.target.wants/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mgr.smithi037.gtwkck.service → /etc/systemd/system/ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@.service. 2024-04-15T22:49:54.064 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:53 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1233739722' entity='client.admin' 2024-04-15T22:49:54.491 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T22:49:54.491 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-04-15T22:49:54.499 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T22:49:54.499 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-04-15T22:49:54.499 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-04-15T22:49:54.499 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to enable service . firewalld.service is not available 2024-04-15T22:49:54.508 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T22:49:54.508 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-04-15T22:49:54.516 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T22:49:54.516 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-04-15T22:49:54.517 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-04-15T22:49:54.517 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-15T22:49:54.517 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr to start... 2024-04-15T22:49:54.517 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr... 2024-04-15T22:49:55.282 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:55 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2015696837' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:49:55.519 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:55.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:49:55.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "60ff499a-fb7a-11ee-bc8f-c7b262605968", 2024-04-15T22:49:55.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-04-15T22:49:55.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-15T22:49:55.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "smithi037" 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-15T22:49:55.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:55.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-15T22:49:55.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-15T22:49:55.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-15T22:49:55.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-15T22:49:55.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:55.532 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:55.533 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-04-15T22:49:47.692029+0000", 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.918+0000 7f76b75ab700 1 Processor -- start 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.918+0000 7f76b75ab700 1 -- start start 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.918+0000 7f76b75ab700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.918+0000 7f76b75ab700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0104750 con 0x7f76b0103e00 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76b5347700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76b5347700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58188/0 (socket says 172.21.15.37:58188) 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76b5347700 1 -- 172.21.15.37:0/1834582048 learned_addr learned my addr 172.21.15.37:0/1834582048 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76b5347700 1 -- 172.21.15.37:0/1834582048 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b0104890 con 0x7f76b0103e00 2024-04-15T22:49:55.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76b5347700 1 --2- 172.21.15.37:0/1834582048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0104210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f76a40098a0 tx=0x7f76a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b4451f1b7e3238 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:55.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76a3fff700 1 -- 172.21.15.37:0/1834582048 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76a4004180 con 0x7f76b0103e00 2024-04-15T22:49:55.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76a3fff700 1 -- 172.21.15.37:0/1834582048 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f76a40042e0 con 0x7f76b0103e00 2024-04-15T22:49:55.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.919+0000 7f76a3fff700 1 -- 172.21.15.37:0/1834582048 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76a4011a50 con 0x7f76b0103e00 2024-04-15T22:49:55.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 -- 172.21.15.37:0/1834582048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 msgr2=0x7f76b0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:55.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 --2- 172.21.15.37:0/1834582048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0104210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f76a40098a0 tx=0x7f76a4009bb0 comp rx=0 tx=0).stop 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 -- 172.21.15.37:0/1834582048 shutdown_connections 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 --2- 172.21.15.37:0/1834582048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0104210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 -- 172.21.15.37:0/1834582048 >> 172.21.15.37:0/1834582048 conn(0x7f76b00ff7e0 msgr2=0x7f76b0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 -- 172.21.15.37:0/1834582048 shutdown_connections 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.920+0000 7f76b75ab700 1 -- 172.21.15.37:0/1834582048 wait complete. 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b75ab700 1 Processor -- start 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b75ab700 1 -- start start 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b75ab700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b75ab700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0199da0 con 0x7f76b0103e00 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b5347700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b5347700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58192/0 (socket says 172.21.15.37:58192) 2024-04-15T22:49:55.538 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b5347700 1 -- 172.21.15.37:0/2015696837 learned_addr learned my addr 172.21.15.37:0/2015696837 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b5347700 1 -- 172.21.15.37:0/2015696837 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4009550 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.921+0000 7f76b5347700 1 --2- 172.21.15.37:0/2015696837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0199860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76a4000c00 tx=0x7f76a4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76a4003b90 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f76a40248b0 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b0199fa0 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76a401a8e0 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b019a3c0 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f76a402d3f0 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.922+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f76a4019490 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.923+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b0062380 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:54.925+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f76a4024a20 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.000+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f76b019cef0 con 0x7f76b0103e00 2024-04-15T22:49:55.539 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.001+0000 7f76a27fc700 1 -- 172.21.15.37:0/2015696837 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f76a4035090 con 0x7f76b0103e00 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 msgr2=0x7f76b0199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 --2- 172.21.15.37:0/2015696837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0199860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76a4000c00 tx=0x7f76a4011ab0 comp rx=0 tx=0).stop 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 shutdown_connections 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 --2- 172.21.15.37:0/2015696837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b0103e00 0x7f76b0199860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 >> 172.21.15.37:0/2015696837 conn(0x7f76b00ff7e0 msgr2=0x7f76b01005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 shutdown_connections 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:55.002+0000 7f76b75ab700 1 -- 172.21.15.37:0/2015696837 wait complete. 2024-04-15T22:49:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:mgr not available, waiting (1/15)... 2024-04-15T22:49:58.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:49:58 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2667153548' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:49:58.594 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:49:58.594 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "60ff499a-fb7a-11ee-bc8f-c7b262605968", 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-15T22:49:58.595 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "smithi037" 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-15T22:49:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-15T22:49:58.605 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-15T22:49:58.606 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-04-15T22:49:47.692029+0000", 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T22:49:58.607 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.980+0000 7f2ec80bd700 1 Processor -- start 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec80bd700 1 -- start start 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec80bd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec80bd700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0104750 con 0x7f2ec0103e00 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec5e59700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec5e59700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58204/0 (socket says 172.21.15.37:58204) 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec5e59700 1 -- 172.21.15.37:0/1239042383 learned_addr learned my addr 172.21.15.37:0/1239042383 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec5e59700 1 -- 172.21.15.37:0/1239042383 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0104890 con 0x7f2ec0103e00 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.981+0000 7f2ec5e59700 1 --2- 172.21.15.37:0/1239042383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0104210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098a0 tx=0x7f2ebc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9932711d189fab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec4e57700 1 -- 172.21.15.37:0/1239042383 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc004180 con 0x7f2ec0103e00 2024-04-15T22:49:58.608 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec4e57700 1 -- 172.21.15.37:0/1239042383 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2ebc0042e0 con 0x7f2ec0103e00 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec4e57700 1 -- 172.21.15.37:0/1239042383 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc011a50 con 0x7f2ec0103e00 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec80bd700 1 -- 172.21.15.37:0/1239042383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 msgr2=0x7f2ec0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec80bd700 1 --2- 172.21.15.37:0/1239042383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0104210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098a0 tx=0x7f2ebc009bb0 comp rx=0 tx=0).stop 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec80bd700 1 -- 172.21.15.37:0/1239042383 shutdown_connections 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec80bd700 1 --2- 172.21.15.37:0/1239042383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0104210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec80bd700 1 -- 172.21.15.37:0/1239042383 >> 172.21.15.37:0/1239042383 conn(0x7f2ec00ff7e0 msgr2=0x7f2ec0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.982+0000 7f2ec80bd700 1 -- 172.21.15.37:0/1239042383 shutdown_connections 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec80bd700 1 -- 172.21.15.37:0/1239042383 wait complete. 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec80bd700 1 Processor -- start 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec80bd700 1 -- start start 2024-04-15T22:49:58.609 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec80bd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec80bd700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0106cd0 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec5e59700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.983+0000 7f2ec5e59700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58210/0 (socket says 172.21.15.37:58210) 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2ec5e59700 1 -- 172.21.15.37:0/2667153548 learned_addr learned my addr 172.21.15.37:0/2667153548 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2ec5e59700 1 -- 172.21.15.37:0/2667153548 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc009550 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2ec5e59700 1 --2- 172.21.15.37:0/2667153548 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0106790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc000c00 tx=0x7f2ebc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc003b90 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2ebc0248b0 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec0104de0 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec0105200 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ebc01a8e0 con 0x7f2ec0103e00 2024-04-15T22:49:58.610 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2ebc02d3f0 con 0x7f2ec0103e00 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.984+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2ebc019460 con 0x7f2ec0103e00 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.985+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea4005320 con 0x7f2ec0103e00 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:57.987+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2ebc024440 con 0x7f2ec0103e00 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.061+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2ea4005cc0 con 0x7f2ec0103e00 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.062+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2667153548 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f2ebc035090 con 0x7f2ec0103e00 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 msgr2=0x7f2ec0106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 --2- 172.21.15.37:0/2667153548 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0106790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc000c00 tx=0x7f2ebc011ab0 comp rx=0 tx=0).stop 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 shutdown_connections 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 --2- 172.21.15.37:0/2667153548 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec0103e00 0x7f2ec0106790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 >> 172.21.15.37:0/2667153548 conn(0x7f2ec00ff7e0 msgr2=0x7f2ec01005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:49:58.611 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 shutdown_connections 2024-04-15T22:49:58.612 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:49:58.063+0000 7f2ec80bd700 1 -- 172.21.15.37:0/2667153548 wait complete. 2024-04-15T22:49:58.612 INFO:teuthology.orchestra.run.smithi037.stdout:mgr not available, waiting (2/15)... 2024-04-15T22:50:01.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: Activating manager daemon smithi037.gtwkck 2024-04-15T22:50:01.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: mgrmap e2: smithi037.gtwkck(active, starting, since 0.00302553s) 2024-04-15T22:50:01.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr metadata", "who": "smithi037.gtwkck", "id": "smithi037.gtwkck"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: Manager daemon smithi037.gtwkck is now available 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/trash_purge_schedule"}]: dispatch 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:01.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:01 smithi037 ceph-mon[29191]: from='mgr.14100 172.21.15.37:0/2545831567' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:03.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:03 smithi037 ceph-mon[29191]: mgrmap e3: smithi037.gtwkck(active, since 1.00768s) 2024-04-15T22:50:04.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:04 smithi037 ceph-mon[29191]: mgrmap e4: smithi037.gtwkck(active, since 2s) 2024-04-15T22:50:05.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:05 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3157587221' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:50:10.103 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:50:10.103 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsid": "60ff499a-fb7a-11ee-bc8f-c7b262605968", 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "health": { 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 0 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "smithi037" 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:50:10.104 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-15T22:50:10.106 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "restful" 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ], 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "modified": "2024-04-15T22:49:47.692029+0000", 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T22:50:10.107 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout }, 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.035+0000 7f05b32cf700 1 Processor -- start 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.035+0000 7f05b32cf700 1 -- start start 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b32cf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b32cf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac105350 con 0x7f05ac104a00 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b106b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b106b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33680/0 (socket says 172.21.15.37:33680) 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b106b700 1 -- 172.21.15.37:0/3302694446 learned_addr learned my addr 172.21.15.37:0/3302694446 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b106b700 1 -- 172.21.15.37:0/3302694446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac105490 con 0x7f05ac104a00 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05b106b700 1 --2- 172.21.15.37:0/3302694446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac104e10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f059c009e80 tx=0x7f059c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75fa7e5b494a7e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05abfff700 1 -- 172.21.15.37:0/3302694446 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059c00e070 con 0x7f05ac104a00 2024-04-15T22:50:10.108 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.036+0000 7f05abfff700 1 -- 172.21.15.37:0/3302694446 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f059c003e90 con 0x7f05ac104a00 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05abfff700 1 -- 172.21.15.37:0/3302694446 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059c004380 con 0x7f05ac104a00 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 -- 172.21.15.37:0/3302694446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 msgr2=0x7f05ac104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 --2- 172.21.15.37:0/3302694446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac104e10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f059c009e80 tx=0x7f059c00b330 comp rx=0 tx=0).stop 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 -- 172.21.15.37:0/3302694446 shutdown_connections 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 --2- 172.21.15.37:0/3302694446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac104e10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 -- 172.21.15.37:0/3302694446 >> 172.21.15.37:0/3302694446 conn(0x7f05ac100070 msgr2=0x7f05ac102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 -- 172.21.15.37:0/3302694446 shutdown_connections 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.037+0000 7f05b32cf700 1 -- 172.21.15.37:0/3302694446 wait complete. 2024-04-15T22:50:10.109 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b32cf700 1 Processor -- start 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b32cf700 1 -- start start 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b32cf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b32cf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac199d20 con 0x7f05ac104a00 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b106b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b106b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac1997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33694/0 (socket says 172.21.15.37:33694) 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b106b700 1 -- 172.21.15.37:0/3157587221 learned_addr learned my addr 172.21.15.37:0/3157587221 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:10.110 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.038+0000 7f05b106b700 1 -- 172.21.15.37:0/3157587221 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c009b30 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.039+0000 7f05b106b700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac1997e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f059c00b7b0 tx=0x7f059c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.039+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059c00e040 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.039+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f059c003c70 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.039+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac199f20 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.039+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059c01b760 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.039+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac19a3a0 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.040+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f059c019040 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.040+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac062380 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.040+0000 7f05aa7fc700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f0594038360 0x7f059403a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.040+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f059c04bd50 con 0x7f05ac104a00 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.040+0000 7f05b086a700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f0594038360 0x7f059403a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.040+0000 7f05b086a700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f0594038360 0x7f059403a810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f05a0006fd0 tx=0x7f05a0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:10.111 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.045+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f059c023d80 con 0x7f05ac104a00 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.329+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f05ac19d1a0 con 0x7f05ac104a00 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.329+0000 7f05aa7fc700 1 -- 172.21.15.37:0/3157587221 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f059c01f090 con 0x7f05ac104a00 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f0594038360 msgr2=0x7f059403a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f0594038360 0x7f059403a810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f05a0006fd0 tx=0x7f05a0006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 msgr2=0x7f05ac1997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac1997e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f059c00b7b0 tx=0x7f059c004430 comp rx=0 tx=0).stop 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 shutdown_connections 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f0594038360 0x7f059403a810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 --2- 172.21.15.37:0/3157587221 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05ac104a00 0x7f05ac1997e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 >> 172.21.15.37:0/3157587221 conn(0x7f05ac100070 msgr2=0x7f05ac074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 shutdown_connections 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:04.331+0000 7f05b32cf700 1 -- 172.21.15.37:0/3157587221 wait complete. 2024-04-15T22:50:10.112 INFO:teuthology.orchestra.run.smithi037.stdout:mgr is available 2024-04-15T22:50:12.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:12 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1327585595' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T22:50:12.543 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:50:12.543 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [global] 2024-04-15T22:50:12.543 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout fsid = 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:50:12.543 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-15T22:50:12.543 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-15T22:50:12.543 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout [osd] 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.783+0000 7f2ca8d10700 1 Processor -- start 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.783+0000 7f2ca8d10700 1 -- start start 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.783+0000 7f2ca8d10700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca4105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:12.544 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.783+0000 7f2ca8d10700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca4106460 con 0x7f2ca4105b10 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca4105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca4105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48036/0 (socket says 172.21.15.37:48036) 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca259c700 1 -- 172.21.15.37:0/3436684104 learned_addr learned my addr 172.21.15.37:0/3436684104 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca259c700 1 -- 172.21.15.37:0/3436684104 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca41065a0 con 0x7f2ca4105b10 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca259c700 1 --2- 172.21.15.37:0/3436684104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca4105f20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c940098a0 tx=0x7f2c94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec043b1e63aeb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca159a700 1 -- 172.21.15.37:0/3436684104 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c94004180 con 0x7f2ca4105b10 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca159a700 1 -- 172.21.15.37:0/3436684104 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2c940042e0 con 0x7f2ca4105b10 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.784+0000 7f2ca159a700 1 -- 172.21.15.37:0/3436684104 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c94011a50 con 0x7f2ca4105b10 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 -- 172.21.15.37:0/3436684104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 msgr2=0x7f2ca4105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 --2- 172.21.15.37:0/3436684104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca4105f20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c940098a0 tx=0x7f2c94009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 -- 172.21.15.37:0/3436684104 shutdown_connections 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 --2- 172.21.15.37:0/3436684104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca4105f20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:12.545 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 -- 172.21.15.37:0/3436684104 >> 172.21.15.37:0/3436684104 conn(0x7f2ca4101180 msgr2=0x7f2ca4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 -- 172.21.15.37:0/3436684104 shutdown_connections 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.785+0000 7f2ca8d10700 1 -- 172.21.15.37:0/3436684104 wait complete. 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca8d10700 1 Processor -- start 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca8d10700 1 -- start start 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca8d10700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca8d10700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca407c180 con 0x7f2ca4105b10 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48040/0 (socket says 172.21.15.37:48040) 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca259c700 1 -- 172.21.15.37:0/1327585595 learned_addr learned my addr 172.21.15.37:0/1327585595 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca259c700 1 -- 172.21.15.37:0/1327585595 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c94009550 con 0x7f2ca4105b10 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.786+0000 7f2ca259c700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca407bc40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2c94000c00 tx=0x7f2c94011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.787+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c94003b90 con 0x7f2ca4105b10 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.787+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2c940248b0 con 0x7f2ca4105b10 2024-04-15T22:50:12.546 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.787+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca407a290 con 0x7f2ca4105b10 2024-04-15T22:50:12.547 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.787+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c9401a8e0 con 0x7f2ca4105b10 2024-04-15T22:50:12.547 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.787+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca407a6b0 con 0x7f2ca4105b10 2024-04-15T22:50:12.547 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.788+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f2c9401ab50 con 0x7f2ca4105b10 2024-04-15T22:50:12.547 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.788+0000 7f2c9b7fe700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f2c900383f0 0x7f2c9003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:12.549 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.788+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2c9404c060 con 0x7f2ca4105b10 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.788+0000 7f2ca1d9b700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f2c900383f0 0x7f2c9003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.788+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca4062380 con 0x7f2ca4105b10 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.788+0000 7f2ca1d9b700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f2c900383f0 0x7f2c9003a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c006fd0 tx=0x7f2c8c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.793+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2c9401ae00 con 0x7f2ca4105b10 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.990+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2ca407b080 con 0x7f2ca4105b10 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.991+0000 7f2c9b7fe700 1 -- 172.21.15.37:0/1327585595 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2c94018b00 con 0x7f2ca4105b10 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f2c900383f0 msgr2=0x7f2c9003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f2c900383f0 0x7f2c9003a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c006fd0 tx=0x7f2c8c006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 msgr2=0x7f2ca407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca407bc40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2c94000c00 tx=0x7f2c94011ab0 comp rx=0 tx=0).stop 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 shutdown_connections 2024-04-15T22:50:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7f2c900383f0 0x7f2c9003a8a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:12.551 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 --2- 172.21.15.37:0/1327585595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ca4105b10 0x7f2ca407bc40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:12.551 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.993+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 >> 172.21.15.37:0/1327585595 conn(0x7f2ca4101180 msgr2=0x7f2ca4101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:12.551 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.994+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 shutdown_connections 2024-04-15T22:50:12.551 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:11.994+0000 7f2ca8d10700 1 -- 172.21.15.37:0/1327585595 wait complete. 2024-04-15T22:50:12.551 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling cephadm module... 2024-04-15T22:50:13.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:13 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2641445792' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-15T22:50:13.856 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.966+0000 7fa0c478e700 1 Processor -- start 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.966+0000 7fa0c478e700 1 -- start start 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.966+0000 7fa0c478e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c478e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc07a810 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c252a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c252a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48042/0 (socket says 172.21.15.37:48042) 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c252a700 1 -- 172.21.15.37:0/578244284 learned_addr learned my addr 172.21.15.37:0/578244284 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c252a700 1 -- 172.21.15.37:0/578244284 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc07a950 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c252a700 1 --2- 172.21.15.37:0/578244284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc07a2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0098a0 tx=0x7fa0ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1047bcad6a71a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c1528700 1 -- 172.21.15.37:0/578244284 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac004180 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c1528700 1 -- 172.21.15.37:0/578244284 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0ac0042e0 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.967+0000 7fa0c1528700 1 -- 172.21.15.37:0/578244284 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac011a50 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 -- 172.21.15.37:0/578244284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 msgr2=0x7fa0bc07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:13.857 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 --2- 172.21.15.37:0/578244284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc07a2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0098a0 tx=0x7fa0ac009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 -- 172.21.15.37:0/578244284 shutdown_connections 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 --2- 172.21.15.37:0/578244284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc07a2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 -- 172.21.15.37:0/578244284 >> 172.21.15.37:0/578244284 conn(0x7fa0bc101180 msgr2=0x7fa0bc103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 -- 172.21.15.37:0/578244284 shutdown_connections 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.968+0000 7fa0c478e700 1 -- 172.21.15.37:0/578244284 wait complete. 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c478e700 1 Processor -- start 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c478e700 1 -- start start 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c478e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c478e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc19bff0 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c252a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c252a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc19bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48052/0 (socket says 172.21.15.37:48052) 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c252a700 1 -- 172.21.15.37:0/2641445792 learned_addr learned my addr 172.21.15.37:0/2641445792 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.969+0000 7fa0c252a700 1 -- 172.21.15.37:0/2641445792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac009550 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.970+0000 7fa0c252a700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc19bab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac000c00 tx=0x7fa0ac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.970+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac003a00 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.970+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0ac003b60 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.970+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac01a800 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.970+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc19c1f0 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.970+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc19c610 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.971+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7fa0ac024440 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.971+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc062380 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.971+0000 7fa0bb7fe700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa0a40383f0 0x7fa0a403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.971+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa0ac04bff0 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.971+0000 7fa0c1d29700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa0a40383f0 0x7fa0a403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.971+0000 7fa0c1d29700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa0a40383f0 0x7fa0a403a8a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0006fd0 tx=0x7fa0b0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:12.976+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa0ac024bb0 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.234+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fa0bc19f790 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.287+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa0ac02d910 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.288+0000 7fa0bb7fe700 1 -- 172.21.15.37:0/2641445792 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fa0ac04bd50 con 0x7fa0bc07bdd0 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa0a40383f0 msgr2=0x7fa0a403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa0a40383f0 0x7fa0a403a8a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0006fd0 tx=0x7fa0b0006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 msgr2=0x7fa0bc19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc19bab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac000c00 tx=0x7fa0ac0047c0 comp rx=0 tx=0).stop 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 shutdown_connections 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa0a40383f0 0x7fa0a403a8a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 --2- 172.21.15.37:0/2641445792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0bc07bdd0 0x7fa0bc19bab0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 >> 172.21.15.37:0/2641445792 conn(0x7fa0bc101180 msgr2=0x7fa0bc102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 shutdown_connections 2024-04-15T22:50:13.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:13.291+0000 7fa0c478e700 1 -- 172.21.15.37:0/2641445792 wait complete. 2024-04-15T22:50:14.492 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:14 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2641445792' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-15T22:50:14.492 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:14 smithi037 ceph-mon[29191]: mgrmap e5: smithi037.gtwkck(active, since 12s) 2024-04-15T22:50:15.121 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "active_name": "smithi037.gtwkck", 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3e0f55700 1 Processor -- start 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3e0f55700 1 -- start start 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3e0f55700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc104870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3e0f55700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0744b0 con 0x7fa3dc104460 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3da59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc104870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3da59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc104870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48084/0 (socket says 172.21.15.37:48084) 2024-04-15T22:50:15.122 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.288+0000 7fa3da59c700 1 -- 172.21.15.37:0/3699894602 learned_addr learned my addr 172.21.15.37:0/3699894602 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.289+0000 7fa3da59c700 1 -- 172.21.15.37:0/3699894602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc0745f0 con 0x7fa3dc104460 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.289+0000 7fa3da59c700 1 --2- 172.21.15.37:0/3699894602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc104870 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0098a0 tx=0x7fa3cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f182acabae2575f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.289+0000 7fa3d959a700 1 -- 172.21.15.37:0/3699894602 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3cc004180 con 0x7fa3dc104460 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.289+0000 7fa3d959a700 1 -- 172.21.15.37:0/3699894602 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa3cc0042e0 con 0x7fa3dc104460 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.289+0000 7fa3d959a700 1 -- 172.21.15.37:0/3699894602 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3cc011a50 con 0x7fa3dc104460 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3699894602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 msgr2=0x7fa3dc104870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 --2- 172.21.15.37:0/3699894602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc104870 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0098a0 tx=0x7fa3cc009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3699894602 shutdown_connections 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 --2- 172.21.15.37:0/3699894602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc104870 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3699894602 >> 172.21.15.37:0/3699894602 conn(0x7fa3dc100070 msgr2=0x7fa3dc102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3699894602 shutdown_connections 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.290+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3699894602 wait complete. 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3e0f55700 1 Processor -- start 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3e0f55700 1 -- start start 2024-04-15T22:50:15.123 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3e0f55700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3e0f55700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc19bf70 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3da59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3da59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc19ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48088/0 (socket says 172.21.15.37:48088) 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3da59c700 1 -- 172.21.15.37:0/3885770613 learned_addr learned my addr 172.21.15.37:0/3885770613 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3da59c700 1 -- 172.21.15.37:0/3885770613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc009550 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.291+0000 7fa3da59c700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc19ba30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc000c00 tx=0x7fa3cc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.292+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3cc003b90 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.292+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc19c170 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.292+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa3cc0248b0 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.292+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3cc01a8e0 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.292+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc19c590 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.293+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc195270 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.293+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa3cc024a20 con 0x7fa3dc104460 2024-04-15T22:50:15.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.293+0000 7fa3cb7fe700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 0x7fa3c403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.293+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa3cc04c0d0 con 0x7fa3dc104460 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.293+0000 7fa3d9d9b700 1 -- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 msgr2=0x7fa3c403a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.293+0000 7fa3d9d9b700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 0x7fa3c403a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.298+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa3cc024440 con 0x7fa3dc104460 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.494+0000 7fa3d9d9b700 1 -- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 msgr2=0x7fa3c403a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.494+0000 7fa3d9d9b700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 0x7fa3c403a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.555+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa3dc0623b0 con 0x7fa3dc104460 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.555+0000 7fa3cb7fe700 1 -- 172.21.15.37:0/3885770613 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa3cc018350 con 0x7fa3dc104460 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 msgr2=0x7fa3c403a8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 0x7fa3c403a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 msgr2=0x7fa3dc19ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc19ba30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc000c00 tx=0x7fa3cc011ab0 comp rx=0 tx=0).stop 2024-04-15T22:50:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 shutdown_connections 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fa3c40383f0 0x7fa3c403a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 --2- 172.21.15.37:0/3885770613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3dc104460 0x7fa3dc19ba30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 >> 172.21.15.37:0/3885770613 conn(0x7fa3dc100070 msgr2=0x7fa3dc190020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 shutdown_connections 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:14.557+0000 7fa3e0f55700 1 -- 172.21.15.37:0/3885770613 wait complete. 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for the mgr to restart... 2024-04-15T22:50:15.126 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr epoch 5... 2024-04-15T22:50:15.382 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:15 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3885770613' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T22:50:20.571 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: Active manager daemon smithi037.gtwkck restarted 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: Activating manager daemon smithi037.gtwkck 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: osdmap e2: 0 total, 0 up, 0 in 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: mgrmap e6: smithi037.gtwkck(active, starting, since 0.00341029s) 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr metadata", "who": "smithi037.gtwkck", "id": "smithi037.gtwkck"}]: dispatch 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:50:20.572 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:50:20.573 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:50:20.573 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: Manager daemon smithi037.gtwkck is now available 2024-04-15T22:50:20.573 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:20.573 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:20.573 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:20.573 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:20.574 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:20.574 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:50:20.574 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:20 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/trash_purge_schedule"}]: dispatch 2024-04-15T22:50:21.393 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:21 smithi037 ceph-mon[29191]: Found migration_current of "None". Setting to last migration. 2024-04-15T22:50:21.393 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:21 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:21.394 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:21 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:21.394 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:21 smithi037 ceph-mon[29191]: mgrmap e7: smithi037.gtwkck(active, since 1.00549s) 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.486+0000 7fcc1c0c2700 1 Processor -- start 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc1c0c2700 1 -- start start 2024-04-15T22:50:21.671 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc1c0c2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc1c0c2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc14106460 con 0x7fcc14105b10 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc19e5e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc19e5e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48098/0 (socket says 172.21.15.37:48098) 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc19e5e700 1 -- 172.21.15.37:0/3478064231 learned_addr learned my addr 172.21.15.37:0/3478064231 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc19e5e700 1 -- 172.21.15.37:0/3478064231 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc141065a0 con 0x7fcc14105b10 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.487+0000 7fcc19e5e700 1 --2- 172.21.15.37:0/3478064231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14105f20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcc100098a0 tx=0x7fcc10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f7ce6d3644cc28 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.488+0000 7fcc18e5c700 1 -- 172.21.15.37:0/3478064231 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc10004180 con 0x7fcc14105b10 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.488+0000 7fcc18e5c700 1 -- 172.21.15.37:0/3478064231 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc100042e0 con 0x7fcc14105b10 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.488+0000 7fcc18e5c700 1 -- 172.21.15.37:0/3478064231 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc10011a50 con 0x7fcc14105b10 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.488+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3478064231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 msgr2=0x7fcc14105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.488+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3478064231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14105f20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcc100098a0 tx=0x7fcc10009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.488+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3478064231 shutdown_connections 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3478064231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14105f20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3478064231 >> 172.21.15.37:0/3478064231 conn(0x7fcc14101160 msgr2=0x7fcc14103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3478064231 shutdown_connections 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3478064231 wait complete. 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 Processor -- start 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 -- start start 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.489+0000 7fcc1c0c2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc14199d70 con 0x7fcc14105b10 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc19e5e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc19e5e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48104/0 (socket says 172.21.15.37:48104) 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc19e5e700 1 -- 172.21.15.37:0/3463804180 learned_addr learned my addr 172.21.15.37:0/3463804180 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc19e5e700 1 -- 172.21.15.37:0/3463804180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10009550 con 0x7fcc14105b10 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc19e5e700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14199830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcc100119e0 tx=0x7fcc10011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc10003b90 con 0x7fcc14105b10 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc1001a8a0 con 0x7fcc14105b10 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc14199f70 con 0x7fcc14105b10 2024-04-15T22:50:21.673 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc10019930 con 0x7fcc14105b10 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.490+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1419a390 con 0x7fcc14105b10 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.491+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fcc10019ba0 con 0x7fcc14105b10 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.491+0000 7fcc0affd700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.491+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 --> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc14109fe0 con 0x7fcc00038450 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.491+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcc1004c690 con 0x7fcc14105b10 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.491+0000 7fcc1965d700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 msgr2=0x7fcc0003a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.491+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.692+0000 7fcc1965d700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 msgr2=0x7fcc0003a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:15.692+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:16.093+0000 7fcc1965d700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 msgr2=0x7fcc0003a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:16.093+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:16.894+0000 7fcc1965d700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 msgr2=0x7fcc0003a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:16.894+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-15T22:50:21.674 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:18.495+0000 7fcc1965d700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 msgr2=0x7fcc0003a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/202293977 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:18.495+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:20.084+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mgrmap(e 6) v1 ==== 44851+0+0 (secure 0 0 0) 0x7fcc1002d440 con 0x7fcc14105b10 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:20.084+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 msgr2=0x7fcc0003a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:20.084+0000 7fcc0affd700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.085+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7fcc1001a430 con 0x7fcc14105b10 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.085+0000 7fcc0affd700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fcc0003bc30 0x7fcc0003e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.086+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc0003e550 con 0x7fcc0003bc30 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.086+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fcc0003bc30 0x7fcc0003e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.086+0000 7fcc1965d700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fcc0003bc30 0x7fcc0003e010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcc04003a10 tx=0x7fcc040092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.087+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcc0003e550 con 0x7fcc0003bc30 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.093+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcc1419ab70 con 0x7fcc0003bc30 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc0affd700 1 -- 172.21.15.37:0/3463804180 <== mgr.14118 v2:172.21.15.37:6800/85756762 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcc1419ab70 con 0x7fcc0003bc30 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fcc0003bc30 msgr2=0x7fcc0003e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fcc0003bc30 0x7fcc0003e010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcc04003a10 tx=0x7fcc040092b0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.675 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 msgr2=0x7fcc14199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14199830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcc100119e0 tx=0x7fcc10011ac0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 shutdown_connections 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fcc0003bc30 0x7fcc0003e010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc14105b10 0x7fcc14199830 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 --2- 172.21.15.37:0/3463804180 >> [v2:172.21.15.37:6800/202293977,v1:172.21.15.37:6801/202293977] conn(0x7fcc00038450 0x7fcc0003a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 >> 172.21.15.37:0/3463804180 conn(0x7fcc14101160 msgr2=0x7fcc14101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 shutdown_connections 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:21.094+0000 7fcc1c0c2700 1 -- 172.21.15.37:0/3463804180 wait complete. 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:mgr epoch 5 is available 2024-04-15T22:50:21.676 INFO:teuthology.orchestra.run.smithi037.stdout:Setting orchestrator backend to cephadm... 2024-04-15T22:50:22.308 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:22 smithi037 ceph-mon[29191]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T22:50:22.308 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:22 smithi037 ceph-mon[29191]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T22:50:22.308 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:22 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.086+0000 7f3dc16f7700 1 Processor -- start 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.086+0000 7f3dc16f7700 1 -- start start 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dc16f7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc106420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dc16f7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc106960 con 0x7f3dbc106010 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dbaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc106420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dbaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc106420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51604/0 (socket says 172.21.15.37:51604) 2024-04-15T22:50:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dbaffd700 1 -- 172.21.15.37:0/2003363745 learned_addr learned my addr 172.21.15.37:0/2003363745 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dbaffd700 1 -- 172.21.15.37:0/2003363745 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc106aa0 con 0x7f3dbc106010 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3dbaffd700 1 --2- 172.21.15.37:0/2003363745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc106420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0098a0 tx=0x7f3dac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd69ff1dfe476c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3db9ffb700 1 -- 172.21.15.37:0/2003363745 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac004180 con 0x7f3dbc106010 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3db9ffb700 1 -- 172.21.15.37:0/2003363745 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3dac0042e0 con 0x7f3dbc106010 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.087+0000 7f3db9ffb700 1 -- 172.21.15.37:0/2003363745 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac011a50 con 0x7f3dbc106010 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 -- 172.21.15.37:0/2003363745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 msgr2=0x7f3dbc106420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 --2- 172.21.15.37:0/2003363745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc106420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0098a0 tx=0x7f3dac009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 -- 172.21.15.37:0/2003363745 shutdown_connections 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 --2- 172.21.15.37:0/2003363745 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc106420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 -- 172.21.15.37:0/2003363745 >> 172.21.15.37:0/2003363745 conn(0x7f3dbc0ff7e0 msgr2=0x7f3dbc101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 -- 172.21.15.37:0/2003363745 shutdown_connections 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.088+0000 7f3dc16f7700 1 -- 172.21.15.37:0/2003363745 wait complete. 2024-04-15T22:50:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dc16f7700 1 Processor -- start 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dc16f7700 1 -- start start 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dc16f7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc1047d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dc16f7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc19c560 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dbaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc1047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dbaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc1047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51620/0 (socket says 172.21.15.37:51620) 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dbaffd700 1 -- 172.21.15.37:0/1032813912 learned_addr learned my addr 172.21.15.37:0/1032813912 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.089+0000 7f3dbaffd700 1 -- 172.21.15.37:0/1032813912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac009550 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.090+0000 7f3dbaffd700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc1047d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0119e0 tx=0x7f3dac011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.090+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac003b90 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.090+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc19c6a0 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.090+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3dac01a8a0 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.090+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac019930 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.090+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc19cac0 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.091+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7f3dac019bf0 con 0x7f3dbc106010 2024-04-15T22:50:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.091+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc0623f0 con 0x7f3dbc106010 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.091+0000 7f3db3fff700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f3da8038290 0x7f3da803a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.091+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3dac04c700 con 0x7f3dbc106010 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.091+0000 7f3dba7fc700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f3da8038290 0x7f3da803a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.091+0000 7f3dba7fc700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f3da8038290 0x7f3da803a740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3da4006fd0 tx=0x7f3da4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.096+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3dbc0623f0 con 0x7f3dbc106010 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.138+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f3dac01aa10 con 0x7f3dbc106010 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.310+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dbc062380 con 0x7f3da8038290 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.316+0000 7f3db3fff700 1 -- 172.21.15.37:0/1032813912 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3dbc062380 con 0x7f3da8038290 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f3da8038290 msgr2=0x7f3da803a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f3da8038290 0x7f3da803a740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3da4006fd0 tx=0x7f3da4006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 msgr2=0x7f3dbc1047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc1047d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0119e0 tx=0x7f3dac011ac0 comp rx=0 tx=0).stop 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 shutdown_connections 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f3da8038290 0x7f3da803a740 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 --2- 172.21.15.37:0/1032813912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dbc106010 0x7f3dbc1047d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 >> 172.21.15.37:0/1032813912 conn(0x7f3dbc0ff7e0 msgr2=0x7f3dbc1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 shutdown_connections 2024-04-15T22:50:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:22.319+0000 7f3dc16f7700 1 -- 172.21.15.37:0/1032813912 wait complete. 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:21] ENGINE Bus STARTING 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:21] ENGINE Serving on http://172.21.15.37:8765 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:21] ENGINE Serving on https://172.21.15.37:7150 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:21] ENGINE Bus STARTED 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: mgrmap e8: smithi037.gtwkck(active, since 2s) 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:23.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:23 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-15T22:50:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f803a4e8700 1 Processor -- start 2024-04-15T22:50:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f803a4e8700 1 -- start start 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f803a4e8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f803a4e8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034105360 con 0x7f8034104a10 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f8033fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f8033fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51624/0 (socket says 172.21.15.37:51624) 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.344+0000 7f8033fff700 1 -- 172.21.15.37:0/618672800 learned_addr learned my addr 172.21.15.37:0/618672800 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.345+0000 7f8033fff700 1 -- 172.21.15.37:0/618672800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80341054a0 con 0x7f8034104a10 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.345+0000 7f8033fff700 1 --2- 172.21.15.37:0/618672800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034104e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f80240098a0 tx=0x7f8024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45ce6aaace7b993 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.345+0000 7f8032ffd700 1 -- 172.21.15.37:0/618672800 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8024004180 con 0x7f8034104a10 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.345+0000 7f8032ffd700 1 -- 172.21.15.37:0/618672800 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80240042e0 con 0x7f8034104a10 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.345+0000 7f8032ffd700 1 -- 172.21.15.37:0/618672800 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8024011a50 con 0x7f8034104a10 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.345+0000 7f803a4e8700 1 -- 172.21.15.37:0/618672800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 msgr2=0x7f8034104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 --2- 172.21.15.37:0/618672800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034104e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f80240098a0 tx=0x7f8024009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 -- 172.21.15.37:0/618672800 shutdown_connections 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 --2- 172.21.15.37:0/618672800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034104e20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 -- 172.21.15.37:0/618672800 >> 172.21.15.37:0/618672800 conn(0x7f8034100060 msgr2=0x7f8034102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 -- 172.21.15.37:0/618672800 shutdown_connections 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 -- 172.21.15.37:0/618672800 wait complete. 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 Processor -- start 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.346+0000 7f803a4e8700 1 -- start start 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f803a4e8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f803a4e8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034199d70 con 0x7f8034104a10 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f8033fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f8033fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51630/0 (socket says 172.21.15.37:51630) 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f8033fff700 1 -- 172.21.15.37:0/3434420665 learned_addr learned my addr 172.21.15.37:0/3434420665 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f8033fff700 1 -- 172.21.15.37:0/3434420665 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024009550 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f8033fff700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034199830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8024000c00 tx=0x7f8024011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8024003b90 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80240248b0 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8034199f70 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802401a8e0 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.347+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803419a390 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.348+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f802401ab50 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.348+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8034062380 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.349+0000 7f80317fa700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f801c038440 0x7f801c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.349+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f802404c400 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.349+0000 7f80337fe700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f801c038440 0x7f801c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.349+0000 7f80337fe700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f801c038440 0x7f801c03a8f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8028006fd0 tx=0x7f8028006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.354+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f802401ae00 con 0x7f8034104a10 2024-04-15T22:50:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.555+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f8034102200 con 0x7f801c038440 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.555+0000 7f80317fa700 1 -- 172.21.15.37:0/3434420665 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f8034102200 con 0x7f801c038440 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f801c038440 msgr2=0x7f801c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f801c038440 0x7f801c03a8f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8028006fd0 tx=0x7f8028006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 msgr2=0x7f8034199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034199830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8024000c00 tx=0x7f8024011ab0 comp rx=0 tx=0).stop 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 shutdown_connections 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f801c038440 0x7f801c03a8f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 --2- 172.21.15.37:0/3434420665 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8034104a10 0x7f8034199830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 >> 172.21.15.37:0/3434420665 conn(0x7f8034100060 msgr2=0x7f8034079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 shutdown_connections 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:23.558+0000 7f803a4e8700 1 -- 172.21.15.37:0/3434420665 wait complete. 2024-04-15T22:50:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:Generating ssh key... 2024-04-15T22:50:24.745 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:24 smithi037 ceph-mon[29191]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:25.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f466bc49700 1 Processor -- start 2024-04-15T22:50:25.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f466bc49700 1 -- start start 2024-04-15T22:50:25.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f466bc49700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:25.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f466bc49700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466407a830 con 0x7f466407bdf0 2024-04-15T22:50:25.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f46699e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:25.497 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f46699e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51634/0 (socket says 172.21.15.37:51634) 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f46699e5700 1 -- 172.21.15.37:0/2735802267 learned_addr learned my addr 172.21.15.37:0/2735802267 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.536+0000 7f46699e5700 1 -- 172.21.15.37:0/2735802267 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466407a970 con 0x7f466407bdf0 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.537+0000 7f46699e5700 1 --2- 172.21.15.37:0/2735802267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466407a2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46600098a0 tx=0x7f4660009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d315042045333be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.537+0000 7f46689e3700 1 -- 172.21.15.37:0/2735802267 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4660004180 con 0x7f466407bdf0 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.537+0000 7f46689e3700 1 -- 172.21.15.37:0/2735802267 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f46600042e0 con 0x7f466407bdf0 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.537+0000 7f46689e3700 1 -- 172.21.15.37:0/2735802267 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4660011a50 con 0x7f466407bdf0 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.537+0000 7f466bc49700 1 -- 172.21.15.37:0/2735802267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 msgr2=0x7f466407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.537+0000 7f466bc49700 1 --2- 172.21.15.37:0/2735802267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466407a2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46600098a0 tx=0x7f4660009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 -- 172.21.15.37:0/2735802267 shutdown_connections 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 --2- 172.21.15.37:0/2735802267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466407a2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 -- 172.21.15.37:0/2735802267 >> 172.21.15.37:0/2735802267 conn(0x7f4664101180 msgr2=0x7f4664103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 -- 172.21.15.37:0/2735802267 shutdown_connections 2024-04-15T22:50:25.498 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 -- 172.21.15.37:0/2735802267 wait complete. 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 Processor -- start 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 -- start start 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.538+0000 7f466bc49700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466419bfb0 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f46699e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f46699e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466419ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51636/0 (socket says 172.21.15.37:51636) 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f46699e5700 1 -- 172.21.15.37:0/2901622909 learned_addr learned my addr 172.21.15.37:0/2901622909 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f46699e5700 1 -- 172.21.15.37:0/2901622909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4660009550 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f46699e5700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466419ba70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4660006af0 tx=0x7f4660011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4660003b90 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f466001a8a0 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466419c1b0 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4660019880 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.539+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466419c5d0 con 0x7f466407bdf0 2024-04-15T22:50:25.499 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.540+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f4660019b40 con 0x7f466407bdf0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.540+0000 7f465affd700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f46500383b0 0x7f465003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.540+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f466004c6f0 con 0x7f466407bdf0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.540+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4664062380 con 0x7f466407bdf0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.540+0000 7f46691e4700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f46500383b0 0x7f465003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.541+0000 7f46691e4700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f46500383b0 0x7f465003a860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4654006fd0 tx=0x7f4654006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.545+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f466002c3a0 con 0x7f466407bdf0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.747+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4664103440 con 0x7f46500383b0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.982+0000 7f465affd700 1 -- 172.21.15.37:0/2901622909 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4664103440 con 0x7f46500383b0 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f46500383b0 msgr2=0x7f465003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f46500383b0 0x7f465003a860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4654006fd0 tx=0x7f4654006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 msgr2=0x7f466419ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466419ba70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4660006af0 tx=0x7f4660011a90 comp rx=0 tx=0).stop 2024-04-15T22:50:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 shutdown_connections 2024-04-15T22:50:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f46500383b0 0x7f465003a860 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 --2- 172.21.15.37:0/2901622909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f466407bdf0 0x7f466419ba70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 >> 172.21.15.37:0/2901622909 conn(0x7f4664101180 msgr2=0x7f4664102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 shutdown_connections 2024-04-15T22:50:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:24.984+0000 7f466bc49700 1 -- 172.21.15.37:0/2901622909 wait complete. 2024-04-15T22:50:26.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:25 smithi037 ceph-mon[29191]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:26.185 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:25 smithi037 ceph-mon[29191]: Generating ssh key... 2024-04-15T22:50:26.185 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:25 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:26.185 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:25 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:26.771 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlmzEVRQ59AGuVSQJDhl/p2+9m1H/5sowcd707oL28VLg0B43YGXR9QyM5V9VryBxROMf7eD6kOZvrMs1NGdH3pJ4YZHl314FlxZuFnZDsKGslZ1Cb+4WqZrmlkEs9SSz6ItlI9pDD1J4kRztHXGGJfRG/n+kczP9oMrFuMkGkjyEFgvGeYD+REz4L9ptsX/A1//H9JBs1TRQNQMFhbiIB7m9zDBUU4EkDad0TaQ9EkFcVlm3lbdNELnCMyE1ZbUync8cyffCuQA5Jr3wRMrFVEJVhTjAX2XCCFoGUAiMfK/6lUcdaHewpvAxXffdIokl6jKBhxhV4X9DUHmPuubCTdE1VhjvRwboiU8a5PCdh3Jzc6ai5/TE43g2ulyPSi3+5bwt++HSNwGbPmL5BU5pr3zJKgaPG9+ZwUsrUzIDJWe7omktLanE4tnUBbMTHvE8K3oeQKsbzttc/RtTDsZsJnSZFg/Eqe5Na3Ps4ZzPMNsvfhtHaLgAAHNHwfcCTs1c= ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.962+0000 7fb73363b700 1 Processor -- start 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.962+0000 7fb73363b700 1 -- start start 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb73363b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c104590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb73363b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c104ad0 con 0x7fb72c104180 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb7313d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb7313d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c104590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51650/0 (socket says 172.21.15.37:51650) 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb7313d7700 1 -- 172.21.15.37:0/2367823655 learned_addr learned my addr 172.21.15.37:0/2367823655 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb7313d7700 1 -- 172.21.15.37:0/2367823655 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c104c10 con 0x7fb72c104180 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb7313d7700 1 --2- 172.21.15.37:0/2367823655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c104590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00af90 tx=0x7fb71c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdc952a98193339 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb72bfff700 1 -- 172.21.15.37:0/2367823655 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71c00f070 con 0x7fb72c104180 2024-04-15T22:50:26.772 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb72bfff700 1 -- 172.21.15.37:0/2367823655 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb71c003e90 con 0x7fb72c104180 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.963+0000 7fb72bfff700 1 -- 172.21.15.37:0/2367823655 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71c004340 con 0x7fb72c104180 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 -- 172.21.15.37:0/2367823655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 msgr2=0x7fb72c104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 --2- 172.21.15.37:0/2367823655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c104590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00af90 tx=0x7fb71c00c330 comp rx=0 tx=0).stop 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 -- 172.21.15.37:0/2367823655 shutdown_connections 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 --2- 172.21.15.37:0/2367823655 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c104590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 -- 172.21.15.37:0/2367823655 >> 172.21.15.37:0/2367823655 conn(0x7fb72c0ff810 msgr2=0x7fb72c101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:26.773 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 -- 172.21.15.37:0/2367823655 shutdown_connections 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.964+0000 7fb73363b700 1 -- 172.21.15.37:0/2367823655 wait complete. 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb73363b700 1 Processor -- start 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb73363b700 1 -- start start 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb73363b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb73363b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c199c90 con 0x7fb72c104180 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb7313d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb7313d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51652/0 (socket says 172.21.15.37:51652) 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb7313d7700 1 -- 172.21.15.37:0/874858132 learned_addr learned my addr 172.21.15.37:0/874858132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb7313d7700 1 -- 172.21.15.37:0/874858132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c00ac40 con 0x7fb72c104180 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.965+0000 7fb7313d7700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c199750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0072a0 tx=0x7fb71c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.966+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71c00f070 con 0x7fb72c104180 2024-04-15T22:50:26.774 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.966+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb71c0043a0 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.966+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c199e90 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.966+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71c013940 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.966+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c19a310 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.967+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fb71c01a070 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.967+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c0623f0 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.967+0000 7fb72a7fc700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fb714038400 0x7fb71403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.967+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb71c04c4a0 con 0x7fb72c104180 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.967+0000 7fb730bd6700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fb714038400 0x7fb71403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:26.775 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.967+0000 7fb730bd6700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fb714038400 0x7fb71403a8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb720006fd0 tx=0x7fb720006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:25.972+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb72c0623f0 con 0x7fb72c104180 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.173+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb72c062380 con 0x7fb714038400 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.174+0000 7fb72a7fc700 1 -- 172.21.15.37:0/874858132 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb72c062380 con 0x7fb714038400 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fb714038400 msgr2=0x7fb71403a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fb714038400 0x7fb71403a8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb720006fd0 tx=0x7fb720006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 msgr2=0x7fb72c199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c199750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0072a0 tx=0x7fb71c003d00 comp rx=0 tx=0).stop 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 shutdown_connections 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fb714038400 0x7fb71403a8b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:26.776 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 --2- 172.21.15.37:0/874858132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb72c104180 0x7fb72c199750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:26.777 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 >> 172.21.15.37:0/874858132 conn(0x7fb72c0ff810 msgr2=0x7fb72c18ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:26.777 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 shutdown_connections 2024-04-15T22:50:26.777 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:26.176+0000 7fb73363b700 1 -- 172.21.15.37:0/874858132 wait complete. 2024-04-15T22:50:26.777 INFO:teuthology.orchestra.run.smithi037.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:50:26.777 INFO:teuthology.orchestra.run.smithi037.stdout:Adding key to root@localhost authorized_keys... 2024-04-15T22:50:26.777 INFO:teuthology.orchestra.run.smithi037.stdout:Adding host smithi037... 2024-04-15T22:50:27.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:26 smithi037 ceph-mon[29191]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:28.547 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:28 smithi037 ceph-mon[29191]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi037", "addr": "172.21.15.37", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:28.548 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:28 smithi037 ceph-mon[29191]: Deploying cephadm binary to smithi037 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Added host 'smithi037' with addr '172.21.15.37' 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1daa075700 1 Processor -- start 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1daa075700 1 -- start start 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1daa075700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da4105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1daa075700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da4106460 con 0x7f1da4105b10 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1da37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da4105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1da37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da4105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51664/0 (socket says 172.21.15.37:51664) 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1da37fe700 1 -- 172.21.15.37:0/3141971740 learned_addr learned my addr 172.21.15.37:0/3141971740 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:30.521 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.177+0000 7f1da37fe700 1 -- 172.21.15.37:0/3141971740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da41065a0 con 0x7f1da4105b10 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1da37fe700 1 --2- 172.21.15.37:0/3141971740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da4105f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d940098a0 tx=0x7f1d94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e31b22488a95c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1da27fc700 1 -- 172.21.15.37:0/3141971740 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d94004180 con 0x7f1da4105b10 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1da27fc700 1 -- 172.21.15.37:0/3141971740 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1d940042e0 con 0x7f1da4105b10 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1da27fc700 1 -- 172.21.15.37:0/3141971740 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d94011a50 con 0x7f1da4105b10 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 -- 172.21.15.37:0/3141971740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 msgr2=0x7f1da4105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 --2- 172.21.15.37:0/3141971740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da4105f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d940098a0 tx=0x7f1d94009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 -- 172.21.15.37:0/3141971740 shutdown_connections 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 --2- 172.21.15.37:0/3141971740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da4105f20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 -- 172.21.15.37:0/3141971740 >> 172.21.15.37:0/3141971740 conn(0x7f1da4101180 msgr2=0x7f1da4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 -- 172.21.15.37:0/3141971740 shutdown_connections 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.178+0000 7f1daa075700 1 -- 172.21.15.37:0/3141971740 wait complete. 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.179+0000 7f1daa075700 1 Processor -- start 2024-04-15T22:50:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.179+0000 7f1daa075700 1 -- start start 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.179+0000 7f1daa075700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.179+0000 7f1daa075700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da407c180 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51670/0 (socket says 172.21.15.37:51670) 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da37fe700 1 -- 172.21.15.37:0/1425012602 learned_addr learned my addr 172.21.15.37:0/1425012602 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da37fe700 1 -- 172.21.15.37:0/1425012602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d94009550 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da37fe700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da407bc40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d94000c00 tx=0x7f1d94011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d94003b90 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1d940248b0 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da407a290 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d9401a8e0 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.180+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da407a6b0 con 0x7f1da4105b10 2024-04-15T22:50:30.523 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.181+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1d9401aba0 con 0x7f1da4105b10 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.181+0000 7f1da0ff9700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1d8c0383f0 0x7f1d8c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.181+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1d9404c460 con 0x7f1da4105b10 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.181+0000 7f1da2ffd700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1d8c0383f0 0x7f1d8c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.181+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da4062380 con 0x7f1da4105b10 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.182+0000 7f1da2ffd700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1d8c0383f0 0x7f1d8c03a8a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1d98006fd0 tx=0x7f1d98006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.186+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1d940237b0 con 0x7f1da4105b10 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:27.389+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi037", "addr": "172.21.15.37", "target": ["mon-mgr", ""]}) v1 -- 0x7f1da407b0b0 con 0x7f1d8c0383f0 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.990+0000 7f1da0ff9700 1 -- 172.21.15.37:0/1425012602 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f1da407b0b0 con 0x7f1d8c0383f0 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1d8c0383f0 msgr2=0x7f1d8c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1d8c0383f0 0x7f1d8c03a8a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1d98006fd0 tx=0x7f1d98006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 msgr2=0x7f1da407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da407bc40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d94000c00 tx=0x7f1d94011ab0 comp rx=0 tx=0).stop 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 shutdown_connections 2024-04-15T22:50:30.524 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1d8c0383f0 0x7f1d8c03a8a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:30.525 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 --2- 172.21.15.37:0/1425012602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1da4105b10 0x7f1da407bc40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:30.525 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 >> 172.21.15.37:0/1425012602 conn(0x7f1da4101180 msgr2=0x7f1da4101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:30.525 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 shutdown_connections 2024-04-15T22:50:30.525 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:29.993+0000 7f1daa075700 1 -- 172.21.15.37:0/1425012602 wait complete. 2024-04-15T22:50:30.525 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying mon service with default placement... 2024-04-15T22:50:31.168 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:30 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:31.168 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:30 smithi037 ceph-mon[29191]: Added host smithi037 2024-04-15T22:50:31.169 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:30 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.908+0000 7f8876f4b700 1 Processor -- start 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.908+0000 7f8876f4b700 1 -- start start 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.908+0000 7f8876f4b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.908+0000 7f8876f4b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870105360 con 0x7f8870104a10 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f8874ce7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f8874ce7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40286/0 (socket says 172.21.15.37:40286) 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f8874ce7700 1 -- 172.21.15.37:0/2970798317 learned_addr learned my addr 172.21.15.37:0/2970798317 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f8874ce7700 1 -- 172.21.15.37:0/2970798317 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88701054a0 con 0x7f8870104a10 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f8874ce7700 1 --2- 172.21.15.37:0/2970798317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870104e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f88640098a0 tx=0x7f8864009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eceaa0128be5760 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:31.990 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f886f7fe700 1 -- 172.21.15.37:0/2970798317 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8864004180 con 0x7f8870104a10 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f886f7fe700 1 -- 172.21.15.37:0/2970798317 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f88640042e0 con 0x7f8870104a10 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.909+0000 7f886f7fe700 1 -- 172.21.15.37:0/2970798317 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8864011a50 con 0x7f8870104a10 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 -- 172.21.15.37:0/2970798317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 msgr2=0x7f8870104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 --2- 172.21.15.37:0/2970798317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870104e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f88640098a0 tx=0x7f8864009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 -- 172.21.15.37:0/2970798317 shutdown_connections 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 --2- 172.21.15.37:0/2970798317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870104e20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 -- 172.21.15.37:0/2970798317 >> 172.21.15.37:0/2970798317 conn(0x7f8870100060 msgr2=0x7f8870102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 -- 172.21.15.37:0/2970798317 shutdown_connections 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 -- 172.21.15.37:0/2970798317 wait complete. 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 Processor -- start 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.910+0000 7f8876f4b700 1 -- start start 2024-04-15T22:50:31.991 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8876f4b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8876f4b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870199d70 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8874ce7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8874ce7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40290/0 (socket says 172.21.15.37:40290) 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8874ce7700 1 -- 172.21.15.37:0/2060758647 learned_addr learned my addr 172.21.15.37:0/2060758647 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8874ce7700 1 -- 172.21.15.37:0/2060758647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864009550 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8874ce7700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870199830 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8864000c00 tx=0x7f8864011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8864003b90 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f88640248b0 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8870199f70 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.911+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886401a8e0 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.912+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887019a390 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.912+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f8864024a20 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.912+0000 7f886dffb700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f88600383f0 0x7f886003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.913+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f886404c550 con 0x7f8870104a10 2024-04-15T22:50:31.992 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.913+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870062380 con 0x7f8870104a10 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.913+0000 7f886ffff700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f88600383f0 0x7f886003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.913+0000 7f886ffff700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f88600383f0 0x7f886003a8a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f885c006fd0 tx=0x7f885c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:30.918+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8864024db0 con 0x7f8870104a10 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.132+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f8870102200 con 0x7f88600383f0 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.137+0000 7f886dffb700 1 -- 172.21.15.37:0/2060758647 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8870102200 con 0x7f88600383f0 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f88600383f0 msgr2=0x7f886003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f88600383f0 0x7f886003a8a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f885c006fd0 tx=0x7f885c006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 msgr2=0x7f8870199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870199830 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8864000c00 tx=0x7f8864011ab0 comp rx=0 tx=0).stop 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 shutdown_connections 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f88600383f0 0x7f886003a8a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 --2- 172.21.15.37:0/2060758647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8870104a10 0x7f8870199830 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 >> 172.21.15.37:0/2060758647 conn(0x7f8870100060 msgr2=0x7f8870079950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:31.993 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 shutdown_connections 2024-04-15T22:50:31.994 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:31.139+0000 7f8876f4b700 1 -- 172.21.15.37:0/2060758647 wait complete. 2024-04-15T22:50:31.994 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying mgr service with default placement... 2024-04-15T22:50:32.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:32 smithi037 ceph-mon[29191]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:32.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:32 smithi037 ceph-mon[29191]: Saving service mon spec with placement count:5 2024-04-15T22:50:32.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:32 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:33.790 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:33 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:33.791 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:33 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:33.791 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:33 smithi037 ceph-mon[29191]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:33.791 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:33 smithi037 ceph-mon[29191]: Saving service mgr spec with placement count:2 2024-04-15T22:50:33.791 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:33 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdefacd9700 1 Processor -- start 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdefacd9700 1 -- start start 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdefacd9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 0x7fdef41086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdefacd9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4108c10 con 0x7fdef4079cf0 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdef8a75700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 0x7fdef41086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdef8a75700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 0x7fdef41086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40298/0 (socket says 172.21.15.37:40298) 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.026+0000 7fdef8a75700 1 -- 172.21.15.37:0/2135747271 learned_addr learned my addr 172.21.15.37:0/2135747271 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.027+0000 7fdef8a75700 1 -- 172.21.15.37:0/2135747271 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4108d50 con 0x7fdef4079cf0 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.027+0000 7fdef8a75700 1 --2- 172.21.15.37:0/2135747271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 0x7fdef41086d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdee80098a0 tx=0x7fdee8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b96d0eea69f456 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.027+0000 7fdef37fe700 1 -- 172.21.15.37:0/2135747271 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee8004180 con 0x7fdef4079cf0 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.027+0000 7fdef37fe700 1 -- 172.21.15.37:0/2135747271 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdee80042e0 con 0x7fdef4079cf0 2024-04-15T22:50:33.798 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.027+0000 7fdef37fe700 1 -- 172.21.15.37:0/2135747271 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee8011af0 con 0x7fdef4079cf0 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 -- 172.21.15.37:0/2135747271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 msgr2=0x7fdef41086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 --2- 172.21.15.37:0/2135747271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 0x7fdef41086d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdee80098a0 tx=0x7fdee8009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 -- 172.21.15.37:0/2135747271 shutdown_connections 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 --2- 172.21.15.37:0/2135747271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef4079cf0 0x7fdef41086d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 -- 172.21.15.37:0/2135747271 >> 172.21.15.37:0/2135747271 conn(0x7fdef4100060 msgr2=0x7fdef41024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 -- 172.21.15.37:0/2135747271 shutdown_connections 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.028+0000 7fdefacd9700 1 -- 172.21.15.37:0/2135747271 wait complete. 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdefacd9700 1 Processor -- start 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdefacd9700 1 -- start start 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdefacd9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 0x7fdef4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdefacd9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef419a230 con 0x7fdef41998e0 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdef8a75700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 0x7fdef4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdef8a75700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 0x7fdef4199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40302/0 (socket says 172.21.15.37:40302) 2024-04-15T22:50:33.799 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdef8a75700 1 -- 172.21.15.37:0/2786329994 learned_addr learned my addr 172.21.15.37:0/2786329994 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.029+0000 7fdef8a75700 1 -- 172.21.15.37:0/2786329994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee8009550 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.030+0000 7fdef8a75700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 0x7fdef4199cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdee8011670 tx=0x7fdee8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.030+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee80039c0 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.030+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdee8003b20 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.030+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef419a430 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.030+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee80247f0 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.030+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef419d090 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.031+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fdee8024a60 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.031+0000 7fdef1ffb700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fdee4038400 0x7fdee403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.031+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdee804d330 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.031+0000 7fdef3fff700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fdee4038400 0x7fdee403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.031+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef4062380 con 0x7fdef41998e0 2024-04-15T22:50:33.800 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.032+0000 7fdef3fff700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fdee4038400 0x7fdee403a8b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdee0006fd0 tx=0x7fdee0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.036+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdee8024d10 con 0x7fdef41998e0 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.251+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef4079f60 con 0x7fdee4038400 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.256+0000 7fdef1ffb700 1 -- 172.21.15.37:0/2786329994 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdef4079f60 con 0x7fdee4038400 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fdee4038400 msgr2=0x7fdee403a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fdee4038400 0x7fdee403a8b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdee0006fd0 tx=0x7fdee0006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 msgr2=0x7fdef4199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 0x7fdef4199cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdee8011670 tx=0x7fdee8011aa0 comp rx=0 tx=0).stop 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 shutdown_connections 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7fdee4038400 0x7fdee403a8b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 --2- 172.21.15.37:0/2786329994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdef41998e0 0x7fdef4199cf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 >> 172.21.15.37:0/2786329994 conn(0x7fdef4100060 msgr2=0x7fdef41017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 shutdown_connections 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:33.258+0000 7fdefacd9700 1 -- 172.21.15.37:0/2786329994 wait complete. 2024-04-15T22:50:33.801 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying crash service with default placement... 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f157002c700 1 Processor -- start 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f157002c700 1 -- start start 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f157002c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f157002c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1568105350 con 0x7f1568104a00 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f156ddc8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f156ddc8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40312/0 (socket says 172.21.15.37:40312) 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.273+0000 7f156ddc8700 1 -- 172.21.15.37:0/1874946782 learned_addr learned my addr 172.21.15.37:0/1874946782 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.274+0000 7f156ddc8700 1 -- 172.21.15.37:0/1874946782 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568105490 con 0x7f1568104a00 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.274+0000 7f156ddc8700 1 --2- 172.21.15.37:0/1874946782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568104e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15640098a0 tx=0x7f1564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3355db89c83051 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.274+0000 7f156cdc6700 1 -- 172.21.15.37:0/1874946782 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1564004180 con 0x7f1568104a00 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.274+0000 7f156cdc6700 1 -- 172.21.15.37:0/1874946782 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f15640042e0 con 0x7f1568104a00 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.274+0000 7f156cdc6700 1 -- 172.21.15.37:0/1874946782 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1564011af0 con 0x7f1568104a00 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 -- 172.21.15.37:0/1874946782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 msgr2=0x7f1568104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 --2- 172.21.15.37:0/1874946782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568104e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15640098a0 tx=0x7f1564009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 -- 172.21.15.37:0/1874946782 shutdown_connections 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 --2- 172.21.15.37:0/1874946782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568104e10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 -- 172.21.15.37:0/1874946782 >> 172.21.15.37:0/1874946782 conn(0x7f1568100070 msgr2=0x7f1568102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 -- 172.21.15.37:0/1874946782 shutdown_connections 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 -- 172.21.15.37:0/1874946782 wait complete. 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 Processor -- start 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.275+0000 7f157002c700 1 -- start start 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.276+0000 7f157002c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.276+0000 7f157002c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1568199db0 con 0x7f1568104a00 2024-04-15T22:50:35.535 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.276+0000 7f156ddc8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.276+0000 7f156ddc8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40318/0 (socket says 172.21.15.37:40318) 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.276+0000 7f156ddc8700 1 -- 172.21.15.37:0/4246801169 learned_addr learned my addr 172.21.15.37:0/4246801169 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.276+0000 7f156ddc8700 1 -- 172.21.15.37:0/4246801169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1564009550 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.277+0000 7f156ddc8700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568199870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1564006ba0 tx=0x7f1564011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.277+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1564003b00 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.277+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f156401a8e0 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.277+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15640199c0 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.277+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1568199fb0 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.277+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156819a310 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.278+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1564019c30 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.278+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1568062380 con 0x7f1568104a00 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.278+0000 7f155effd700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f15540383f0 0x7f155403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.278+0000 7f156d5c7700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f15540383f0 0x7f155403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:35.536 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.278+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f156404c5a0 con 0x7f1568104a00 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.279+0000 7f156d5c7700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f15540383f0 0x7f155403a8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.283+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f156402d440 con 0x7f1568104a00 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.499+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f1568102210 con 0x7f15540383f0 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.505+0000 7f155effd700 1 -- 172.21.15.37:0/4246801169 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f1568102210 con 0x7f15540383f0 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f15540383f0 msgr2=0x7f155403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f15540383f0 0x7f155403a8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 msgr2=0x7f1568199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568199870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1564006ba0 tx=0x7f1564011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 shutdown_connections 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f15540383f0 0x7f155403a8a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 --2- 172.21.15.37:0/4246801169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1568104a00 0x7f1568199870 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 >> 172.21.15.37:0/4246801169 conn(0x7f1568100070 msgr2=0x7f1568074010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 shutdown_connections 2024-04-15T22:50:35.537 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:34.507+0000 7f157002c700 1 -- 172.21.15.37:0/4246801169 wait complete. 2024-04-15T22:50:35.538 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying ceph-exporter service with default placement... 2024-04-15T22:50:35.539 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:35 smithi037 ceph-mon[29191]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:35.539 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:35 smithi037 ceph-mon[29191]: Saving service crash spec with placement * 2024-04-15T22:50:35.539 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:35 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:35.539 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:35 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:36.748 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-15T22:50:36.748 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.963+0000 7f31aedd3700 1 Processor -- start 2024-04-15T22:50:36.748 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.963+0000 7f31aedd3700 1 -- start start 2024-04-15T22:50:36.748 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.963+0000 7f31aedd3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8108a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.963+0000 7f31aedd3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80744d0 con 0x7f31a8106650 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.963+0000 7f31acb6f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8108a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31acb6f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8108a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40320/0 (socket says 172.21.15.37:40320) 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31acb6f700 1 -- 172.21.15.37:0/2844937674 learned_addr learned my addr 172.21.15.37:0/2844937674 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31acb6f700 1 -- 172.21.15.37:0/2844937674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8074610 con 0x7f31a8106650 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31acb6f700 1 --2- 172.21.15.37:0/2844937674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8108a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f319c0098a0 tx=0x7f319c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1302164790015c51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31a77fe700 1 -- 172.21.15.37:0/2844937674 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319c004180 con 0x7f31a8106650 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31a77fe700 1 -- 172.21.15.37:0/2844937674 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f319c0042e0 con 0x7f31a8106650 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.964+0000 7f31a77fe700 1 -- 172.21.15.37:0/2844937674 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319c011af0 con 0x7f31a8106650 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 -- 172.21.15.37:0/2844937674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 msgr2=0x7f31a8108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 --2- 172.21.15.37:0/2844937674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8108a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f319c0098a0 tx=0x7f319c009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 -- 172.21.15.37:0/2844937674 shutdown_connections 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 --2- 172.21.15.37:0/2844937674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8108a30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 -- 172.21.15.37:0/2844937674 >> 172.21.15.37:0/2844937674 conn(0x7f31a8100070 msgr2=0x7f31a8102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:36.749 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 -- 172.21.15.37:0/2844937674 shutdown_connections 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.965+0000 7f31aedd3700 1 -- 172.21.15.37:0/2844937674 wait complete. 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31aedd3700 1 Processor -- start 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31aedd3700 1 -- start start 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31aedd3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31aedd3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8199d80 con 0x7f31a8106650 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31acb6f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31acb6f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40322/0 (socket says 172.21.15.37:40322) 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31acb6f700 1 -- 172.21.15.37:0/3169470820 learned_addr learned my addr 172.21.15.37:0/3169470820 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31acb6f700 1 -- 172.21.15.37:0/3169470820 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c009550 con 0x7f31a8106650 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.966+0000 7f31acb6f700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8199840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f319c000c00 tx=0x7f319c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.967+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319c003a70 con 0x7f31a8106650 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.967+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f319c003bd0 con 0x7f31a8106650 2024-04-15T22:50:36.750 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.967+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f319c0238c0 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.967+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a8199fe0 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.967+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a819a460 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.968+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f319c023b80 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.968+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a8109b70 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.968+0000 7f31a5ffb700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f31980383f0 0x7f319803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.968+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f319c04c1a0 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.968+0000 7f31a7fff700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f31980383f0 0x7f319803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.969+0000 7f31a7fff700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f31980383f0 0x7f319803a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3194006fd0 tx=0x7f3194006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:35.973+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f319c023e30 con 0x7f31a8106650 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.188+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a8102380 con 0x7f31980383f0 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.193+0000 7f31a5ffb700 1 -- 172.21.15.37:0/3169470820 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f31a8102380 con 0x7f31980383f0 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f31980383f0 msgr2=0x7f319803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f31980383f0 0x7f319803a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3194006fd0 tx=0x7f3194006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:36.751 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 msgr2=0x7f31a8199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8199840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f319c000c00 tx=0x7f319c011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 shutdown_connections 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f31980383f0 0x7f319803a8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 --2- 172.21.15.37:0/3169470820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a8106650 0x7f31a8199840 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 >> 172.21.15.37:0/3169470820 conn(0x7f31a8100070 msgr2=0x7f31a8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 shutdown_connections 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:36.196+0000 7f31aedd3700 1 -- 172.21.15.37:0/3169470820 wait complete. 2024-04-15T22:50:36.752 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying prometheus service with default placement... 2024-04-15T22:50:37.120 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:37 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:37.120 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:37 smithi037 ceph-mon[29191]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:37.120 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:37 smithi037 ceph-mon[29191]: Saving service ceph-exporter spec with placement * 2024-04-15T22:50:37.120 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:37 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:38.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:38 smithi037 ceph-mon[29191]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:38.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:38 smithi037 ceph-mon[29191]: Saving service prometheus spec with placement count:1 2024-04-15T22:50:38.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:38 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.121+0000 7f1c293ae700 1 Processor -- start 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c293ae700 1 -- start start 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c293ae700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c24104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c293ae700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24105360 con 0x7f1c24104a10 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c22ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c24104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:38.653 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c22ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c24104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40326/0 (socket says 172.21.15.37:40326) 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c22ffd700 1 -- 172.21.15.37:0/791360598 learned_addr learned my addr 172.21.15.37:0/791360598 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.122+0000 7f1c22ffd700 1 -- 172.21.15.37:0/791360598 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c241054a0 con 0x7f1c24104a10 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c22ffd700 1 --2- 172.21.15.37:0/791360598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c24104e20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400bb40 tx=0x7f1c1400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c242092ff1cf199 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c21ffb700 1 -- 172.21.15.37:0/791360598 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1400dc00 con 0x7f1c24104a10 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c21ffb700 1 -- 172.21.15.37:0/791360598 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c140044e0 con 0x7f1c24104a10 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c21ffb700 1 -- 172.21.15.37:0/791360598 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14010470 con 0x7f1c24104a10 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 -- 172.21.15.37:0/791360598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 msgr2=0x7f1c24104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 --2- 172.21.15.37:0/791360598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c24104e20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400bb40 tx=0x7f1c1400be50 comp rx=0 tx=0).stop 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 -- 172.21.15.37:0/791360598 shutdown_connections 2024-04-15T22:50:38.654 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 --2- 172.21.15.37:0/791360598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c24104e20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 -- 172.21.15.37:0/791360598 >> 172.21.15.37:0/791360598 conn(0x7f1c24100060 msgr2=0x7f1c24102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 -- 172.21.15.37:0/791360598 shutdown_connections 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.123+0000 7f1c293ae700 1 -- 172.21.15.37:0/791360598 wait complete. 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.124+0000 7f1c293ae700 1 Processor -- start 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.124+0000 7f1c293ae700 1 -- start start 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.124+0000 7f1c293ae700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c2419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.124+0000 7f1c293ae700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2419bfd0 con 0x7f1c24104a10 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c22ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c2419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c22ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c2419ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40336/0 (socket says 172.21.15.37:40336) 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c22ffd700 1 -- 172.21.15.37:0/3280856236 learned_addr learned my addr 172.21.15.37:0/3280856236 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c22ffd700 1 -- 172.21.15.37:0/3280856236 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1400b7f0 con 0x7f1c24104a10 2024-04-15T22:50:38.655 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c22ffd700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c2419ba90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400de50 tx=0x7f1c1400df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14004220 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c14010b90 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2419c1d0 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14021c30 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.125+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2419c590 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.126+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1c140193f0 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.126+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c24195750 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.126+0000 7f1c0bfff700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1c0c0383f0 0x7f1c0c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.127+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1c1404c510 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.127+0000 7f1c227fc700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1c0c0383f0 0x7f1c0c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.127+0000 7f1c227fc700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1c0c0383f0 0x7f1c0c03a8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1c18006fd0 tx=0x7f1c18006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.132+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1c1400e750 con 0x7f1c24104a10 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.347+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c24061190 con 0x7f1c0c0383f0 2024-04-15T22:50:38.656 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.352+0000 7f1c0bfff700 1 -- 172.21.15.37:0/3280856236 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f1c24061190 con 0x7f1c0c0383f0 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.353+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1c0c0383f0 msgr2=0x7f1c0c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.353+0000 7f1c293ae700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1c0c0383f0 0x7f1c0c03a8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1c18006fd0 tx=0x7f1c18006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 msgr2=0x7f1c2419ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c2419ba90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400de50 tx=0x7f1c1400df30 comp rx=0 tx=0).stop 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 shutdown_connections 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1c0c0383f0 0x7f1c0c03a8a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 --2- 172.21.15.37:0/3280856236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c24104a10 0x7f1c2419ba90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 >> 172.21.15.37:0/3280856236 conn(0x7f1c24100060 msgr2=0x7f1c24192890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 shutdown_connections 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:37.354+0000 7f1c293ae700 1 -- 172.21.15.37:0/3280856236 wait complete. 2024-04-15T22:50:38.657 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying grafana service with default placement... 2024-04-15T22:50:39.924 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.085+0000 7ff375529700 1 Processor -- start 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.085+0000 7ff375529700 1 -- start start 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.085+0000 7ff375529700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff370106430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.085+0000 7ff375529700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff370106970 con 0x7ff370106020 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff370106430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff370106430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40342/0 (socket says 172.21.15.37:40342) 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36effd700 1 -- 172.21.15.37:0/2638821502 learned_addr learned my addr 172.21.15.37:0/2638821502 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36effd700 1 -- 172.21.15.37:0/2638821502 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370106ab0 con 0x7ff370106020 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36effd700 1 --2- 172.21.15.37:0/2638821502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff370106430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff360009e80 tx=0x7ff36000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1468052ba0b717a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36dffb700 1 -- 172.21.15.37:0/2638821502 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff36000e070 con 0x7ff370106020 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36dffb700 1 -- 172.21.15.37:0/2638821502 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff36000b8c0 con 0x7ff370106020 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.086+0000 7ff36dffb700 1 -- 172.21.15.37:0/2638821502 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3600044d0 con 0x7ff370106020 2024-04-15T22:50:39.925 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 -- 172.21.15.37:0/2638821502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 msgr2=0x7ff370106430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 --2- 172.21.15.37:0/2638821502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff370106430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff360009e80 tx=0x7ff36000b330 comp rx=0 tx=0).stop 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 -- 172.21.15.37:0/2638821502 shutdown_connections 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 --2- 172.21.15.37:0/2638821502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff370106430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 -- 172.21.15.37:0/2638821502 >> 172.21.15.37:0/2638821502 conn(0x7ff3700ff7e0 msgr2=0x7ff370101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 -- 172.21.15.37:0/2638821502 shutdown_connections 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.087+0000 7ff375529700 1 -- 172.21.15.37:0/2638821502 wait complete. 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff375529700 1 Processor -- start 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff375529700 1 -- start start 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff375529700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff37018fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff375529700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff370190170 con 0x7ff370106020 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff36effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff37018fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff36effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff37018fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40346/0 (socket says 172.21.15.37:40346) 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff36effd700 1 -- 172.21.15.37:0/4127963892 learned_addr learned my addr 172.21.15.37:0/4127963892 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:39.926 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff36effd700 1 -- 172.21.15.37:0/4127963892 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff360009b30 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.088+0000 7ff36effd700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff37018fc30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff360004940 tx=0x7ff360004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.089+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff36000e040 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.089+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff36000bc40 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.089+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37018e280 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.089+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3600126e0 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.089+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37018e700 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.090+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7ff360019040 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.090+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff370062380 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.090+0000 7ff367fff700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7ff35c0383f0 0x7ff35c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.090+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff36004bd60 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.090+0000 7ff36e7fc700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7ff35c0383f0 0x7ff35c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.090+0000 7ff36e7fc700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7ff35c0383f0 0x7ff35c03a8a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff358006fd0 tx=0x7ff358006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.095+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff360011750 con 0x7ff370106020 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.328+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7ff370101940 con 0x7ff35c0383f0 2024-04-15T22:50:39.927 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.333+0000 7ff367fff700 1 -- 172.21.15.37:0/4127963892 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff370101940 con 0x7ff35c0383f0 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7ff35c0383f0 msgr2=0x7ff35c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7ff35c0383f0 0x7ff35c03a8a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff358006fd0 tx=0x7ff358006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 msgr2=0x7ff37018fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff37018fc30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff360004940 tx=0x7ff360004a20 comp rx=0 tx=0).stop 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 shutdown_connections 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7ff35c0383f0 0x7ff35c03a8a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 --2- 172.21.15.37:0/4127963892 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff370106020 0x7ff37018fc30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 >> 172.21.15.37:0/4127963892 conn(0x7ff3700ff7e0 msgr2=0x7ff3701045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 shutdown_connections 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:39.335+0000 7ff375529700 1 -- 172.21.15.37:0/4127963892 wait complete. 2024-04-15T22:50:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying node-exporter service with default placement... 2024-04-15T22:50:40.545 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:40 smithi037 ceph-mon[29191]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:40.545 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:40 smithi037 ceph-mon[29191]: Saving service grafana spec with placement count:1 2024-04-15T22:50:40.545 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:40 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:41.089 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-15T22:50:41.089 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f818d5ee700 1 Processor -- start 2024-04-15T22:50:41.089 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f818d5ee700 1 -- start start 2024-04-15T22:50:41.089 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f818d5ee700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f818d5ee700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8188104750 con 0x7f8188103e00 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f8186ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f8186ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54196/0 (socket says 172.21.15.37:54196) 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.327+0000 7f8186ffd700 1 -- 172.21.15.37:0/3177093555 learned_addr learned my addr 172.21.15.37:0/3177093555 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f8186ffd700 1 -- 172.21.15.37:0/3177093555 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188104890 con 0x7f8188103e00 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f8186ffd700 1 --2- 172.21.15.37:0/3177093555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188104210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f81780098a0 tx=0x7f8178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40deb48336c08001 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f8185ffb700 1 -- 172.21.15.37:0/3177093555 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8178004180 con 0x7f8188103e00 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f8185ffb700 1 -- 172.21.15.37:0/3177093555 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f81780042e0 con 0x7f8188103e00 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f8185ffb700 1 -- 172.21.15.37:0/3177093555 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8178011af0 con 0x7f8188103e00 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f818d5ee700 1 -- 172.21.15.37:0/3177093555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 msgr2=0x7f8188104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f818d5ee700 1 --2- 172.21.15.37:0/3177093555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188104210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f81780098a0 tx=0x7f8178009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f818d5ee700 1 -- 172.21.15.37:0/3177093555 shutdown_connections 2024-04-15T22:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f818d5ee700 1 --2- 172.21.15.37:0/3177093555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188104210 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.328+0000 7f818d5ee700 1 -- 172.21.15.37:0/3177093555 >> 172.21.15.37:0/3177093555 conn(0x7f81880ff7e0 msgr2=0x7f8188101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.329+0000 7f818d5ee700 1 -- 172.21.15.37:0/3177093555 shutdown_connections 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.329+0000 7f818d5ee700 1 -- 172.21.15.37:0/3177093555 wait complete. 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.329+0000 7f818d5ee700 1 Processor -- start 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.329+0000 7f818d5ee700 1 -- start start 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.329+0000 7f818d5ee700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f818d5ee700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8188106cd0 con 0x7f8188103e00 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f8186ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f8186ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54202/0 (socket says 172.21.15.37:54202) 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f8186ffd700 1 -- 172.21.15.37:0/5846639 learned_addr learned my addr 172.21.15.37:0/5846639 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f8186ffd700 1 -- 172.21.15.37:0/5846639 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8178009550 con 0x7f8188103e00 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f8186ffd700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188106790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8178006ba0 tx=0x7f8178011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8178003b00 con 0x7f8188103e00 2024-04-15T22:50:41.091 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f817801a8e0 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8178019990 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8188104de0 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.330+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8188105200 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.331+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f8178019c00 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.331+0000 7f816ffff700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f81700383f0 0x7f817003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.331+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8188062380 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.331+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f817804c570 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.331+0000 7f81867fc700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f81700383f0 0x7f817003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.332+0000 7f81867fc700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f81700383f0 0x7f817003a8a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f817c006fd0 tx=0x7f817c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.336+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f817802d440 con 0x7f8188103e00 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.547+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f8188100cf0 con 0x7f81700383f0 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.552+0000 7f816ffff700 1 -- 172.21.15.37:0/5846639 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f8188100cf0 con 0x7f81700383f0 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f81700383f0 msgr2=0x7f817003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:41.092 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f81700383f0 0x7f817003a8a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f817c006fd0 tx=0x7f817c006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 msgr2=0x7f8188106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188106790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8178006ba0 tx=0x7f8178011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 shutdown_connections 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f81700383f0 0x7f817003a8a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 --2- 172.21.15.37:0/5846639 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8188103e00 0x7f8188106790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 >> 172.21.15.37:0/5846639 conn(0x7f81880ff7e0 msgr2=0x7f81881005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.554+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 shutdown_connections 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:40.555+0000 7f818d5ee700 1 -- 172.21.15.37:0/5846639 wait complete. 2024-04-15T22:50:41.093 INFO:teuthology.orchestra.run.smithi037.stdout:Deploying alertmanager service with default placement... 2024-04-15T22:50:41.509 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:41 smithi037 ceph-mon[29191]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:50:41.509 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:41 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fef68700 1 Processor -- start 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fef68700 1 -- start start 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fef68700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8104e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fef68700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8105340 con 0x7f17f81049f0 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fcd04700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fcd04700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8104e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54216/0 (socket says 172.21.15.37:54216) 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.555+0000 7f17fcd04700 1 -- 172.21.15.37:0/2426591023 learned_addr learned my addr 172.21.15.37:0/2426591023 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.556+0000 7f17fcd04700 1 -- 172.21.15.37:0/2426591023 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f8105480 con 0x7f17f81049f0 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.556+0000 7f17fcd04700 1 --2- 172.21.15.37:0/2426591023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8104e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0098a0 tx=0x7f17ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4510f973a690cba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:42.280 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.556+0000 7f17f77fe700 1 -- 172.21.15.37:0/2426591023 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec004180 con 0x7f17f81049f0 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.556+0000 7f17f77fe700 1 -- 172.21.15.37:0/2426591023 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f17ec0042e0 con 0x7f17f81049f0 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.556+0000 7f17f77fe700 1 -- 172.21.15.37:0/2426591023 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec011af0 con 0x7f17f81049f0 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.556+0000 7f17fef68700 1 -- 172.21.15.37:0/2426591023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 msgr2=0x7f17f8104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 --2- 172.21.15.37:0/2426591023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8104e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0098a0 tx=0x7f17ec009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 -- 172.21.15.37:0/2426591023 shutdown_connections 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 --2- 172.21.15.37:0/2426591023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8104e00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 -- 172.21.15.37:0/2426591023 >> 172.21.15.37:0/2426591023 conn(0x7f17f8100080 msgr2=0x7f17f8102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 -- 172.21.15.37:0/2426591023 shutdown_connections 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 -- 172.21.15.37:0/2426591023 wait complete. 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 Processor -- start 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 -- start start 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.557+0000 7f17fef68700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fef68700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8199d80 con 0x7f17f81049f0 2024-04-15T22:50:42.281 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fcd04700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fcd04700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54218/0 (socket says 172.21.15.37:54218) 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fcd04700 1 -- 172.21.15.37:0/1701751880 learned_addr learned my addr 172.21.15.37:0/1701751880 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fcd04700 1 -- 172.21.15.37:0/1701751880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec009550 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fcd04700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8199840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17ec011ad0 tx=0x7f17ec011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec003ab0 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f17ec003c10 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f8199f80 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec0198e0 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.558+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f819a3a0 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.559+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f17ec019b50 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.559+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8062380 con 0x7f17f81049f0 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.560+0000 7f17f5ffb700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f17e8038440 0x7f17e803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:42.282 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.560+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f17ec04c190 con 0x7f17f81049f0 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.560+0000 7f17f7fff700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f17e8038440 0x7f17e803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.560+0000 7f17f7fff700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f17e8038440 0x7f17e803a8f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006fd0 tx=0x7f17e4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.564+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f17ec010da0 con 0x7f17f81049f0 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.778+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f81021e0 con 0x7f17e8038440 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.783+0000 7f17f5ffb700 1 -- 172.21.15.37:0/1701751880 <== mgr.14118 v2:172.21.15.37:6800/85756762 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f17f81021e0 con 0x7f17e8038440 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f17e8038440 msgr2=0x7f17e803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f17e8038440 0x7f17e803a8f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006fd0 tx=0x7f17e4006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 msgr2=0x7f17f8199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8199840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17ec011ad0 tx=0x7f17ec011bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 shutdown_connections 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f17e8038440 0x7f17e803a8f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 --2- 172.21.15.37:0/1701751880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17f81049f0 0x7f17f8199840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 >> 172.21.15.37:0/1701751880 conn(0x7f17f8100080 msgr2=0x7f17f8079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:42.283 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 shutdown_connections 2024-04-15T22:50:42.284 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:41.786+0000 7f17fef68700 1 -- 172.21.15.37:0/1701751880 wait complete. 2024-04-15T22:50:42.624 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:42 smithi037 ceph-mon[29191]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:42.624 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:42 smithi037 ceph-mon[29191]: Saving service node-exporter spec with placement * 2024-04-15T22:50:42.624 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:42 smithi037 ceph-mon[29191]: from='mgr.14118 172.21.15.37:0/109361207' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:43.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:43 smithi037 ceph-mon[29191]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:43.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:43 smithi037 ceph-mon[29191]: Saving service alertmanager spec with placement count:1 2024-04-15T22:50:43.519 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:43 smithi037 ceph-mon[29191]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:50:43.520 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:43 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2736498501' entity='client.admin' 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.736+0000 7f816bcf2700 1 Processor -- start 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.736+0000 7f816bcf2700 1 -- start start 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f816bcf2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164108a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f816bcf2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8164079d10 con 0x7f8164106660 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8169a8e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164108a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8169a8e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164108a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54220/0 (socket says 172.21.15.37:54220) 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8169a8e700 1 -- 172.21.15.37:0/2785294987 learned_addr learned my addr 172.21.15.37:0/2785294987 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:43.527 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8169a8e700 1 -- 172.21.15.37:0/2785294987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164079e50 con 0x7f8164106660 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8169a8e700 1 --2- 172.21.15.37:0/2785294987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164108a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f81600098a0 tx=0x7f8160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec650f3c0c15849 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8168a8c700 1 -- 172.21.15.37:0/2785294987 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8160004180 con 0x7f8164106660 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8168a8c700 1 -- 172.21.15.37:0/2785294987 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f81600042e0 con 0x7f8164106660 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.737+0000 7f8168a8c700 1 -- 172.21.15.37:0/2785294987 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8160011af0 con 0x7f8164106660 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 -- 172.21.15.37:0/2785294987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 msgr2=0x7f8164108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 --2- 172.21.15.37:0/2785294987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164108a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f81600098a0 tx=0x7f8160009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 -- 172.21.15.37:0/2785294987 shutdown_connections 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 --2- 172.21.15.37:0/2785294987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164108a40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 -- 172.21.15.37:0/2785294987 >> 172.21.15.37:0/2785294987 conn(0x7f8164100080 msgr2=0x7f8164102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 -- 172.21.15.37:0/2785294987 shutdown_connections 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.738+0000 7f816bcf2700 1 -- 172.21.15.37:0/2785294987 wait complete. 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f816bcf2700 1 Processor -- start 2024-04-15T22:50:43.528 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f816bcf2700 1 -- start start 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f816bcf2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164105fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f816bcf2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8164106500 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f8169a8e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164105fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f8169a8e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164105fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54224/0 (socket says 172.21.15.37:54224) 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f8169a8e700 1 -- 172.21.15.37:0/2736498501 learned_addr learned my addr 172.21.15.37:0/2736498501 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.739+0000 7f8169a8e700 1 -- 172.21.15.37:0/2736498501 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8160009550 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.740+0000 7f8169a8e700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164105fc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8160000c00 tx=0x7f8160011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.740+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8160003a70 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.740+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8160003bd0 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.740+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8164104610 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.740+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81600238c0 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.740+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8164104a30 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.741+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f816001a430 con 0x7f8164106660 2024-04-15T22:50:43.529 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.741+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8164109be0 con 0x7f8164106660 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.741+0000 7f815affd700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f8150038440 0x7f815003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.741+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f816004c170 con 0x7f8164106660 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.741+0000 7f816928d700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f8150038440 0x7f815003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.741+0000 7f816928d700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f8150038440 0x7f815003a8f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8154006fd0 tx=0x7f8154006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.746+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8160010560 con 0x7f8164106660 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.943+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f81641054f0 con 0x7f8164106660 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.949+0000 7f815affd700 1 -- 172.21.15.37:0/2736498501 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8160010380 con 0x7f8164106660 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f8150038440 msgr2=0x7f815003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f8150038440 0x7f815003a8f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8154006fd0 tx=0x7f8154006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 msgr2=0x7f8164105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164105fc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8160000c00 tx=0x7f8160011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 shutdown_connections 2024-04-15T22:50:43.530 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f8150038440 0x7f815003a8f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:43.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 --2- 172.21.15.37:0/2736498501 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8164106660 0x7f8164105fc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:43.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 >> 172.21.15.37:0/2736498501 conn(0x7f8164100080 msgr2=0x7f8164101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:43.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 shutdown_connections 2024-04-15T22:50:43.531 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:42.951+0000 7f816bcf2700 1 -- 172.21.15.37:0/2736498501 wait complete. 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.951+0000 7f7344269700 1 Processor -- start 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7344269700 1 -- start start 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7344269700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c1048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7344269700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c079d10 con 0x7f733c104490 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7342005700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c1048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7342005700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c1048a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54232/0 (socket says 172.21.15.37:54232) 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7342005700 1 -- 172.21.15.37:0/2433724043 learned_addr learned my addr 172.21.15.37:0/2433724043 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7342005700 1 -- 172.21.15.37:0/2433724043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c079e50 con 0x7f733c104490 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.952+0000 7f7342005700 1 --2- 172.21.15.37:0/2433724043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c1048a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f73380098a0 tx=0x7f7338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263a1673c6484265 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7341003700 1 -- 172.21.15.37:0/2433724043 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7338004180 con 0x7f733c104490 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7341003700 1 -- 172.21.15.37:0/2433724043 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f73380042e0 con 0x7f733c104490 2024-04-15T22:50:44.684 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7341003700 1 -- 172.21.15.37:0/2433724043 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7338011af0 con 0x7f733c104490 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 -- 172.21.15.37:0/2433724043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 msgr2=0x7f733c1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 --2- 172.21.15.37:0/2433724043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c1048a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f73380098a0 tx=0x7f7338009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 -- 172.21.15.37:0/2433724043 shutdown_connections 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 --2- 172.21.15.37:0/2433724043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c1048a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 -- 172.21.15.37:0/2433724043 >> 172.21.15.37:0/2433724043 conn(0x7f733c100060 msgr2=0x7f733c1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 -- 172.21.15.37:0/2433724043 shutdown_connections 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.953+0000 7f7344269700 1 -- 172.21.15.37:0/2433724043 wait complete. 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7344269700 1 Processor -- start 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7344269700 1 -- start start 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7344269700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7344269700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c19c070 con 0x7f733c104490 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7342005700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7342005700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54242/0 (socket says 172.21.15.37:54242) 2024-04-15T22:50:44.685 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7342005700 1 -- 172.21.15.37:0/811744392 learned_addr learned my addr 172.21.15.37:0/811744392 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.954+0000 7f7342005700 1 -- 172.21.15.37:0/811744392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7338009550 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.955+0000 7f7342005700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c19bb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7338011670 tx=0x7f7338011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.955+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73380039c0 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.955+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7338003b20 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.955+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73380247f0 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.955+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c19c270 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.955+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c19c690 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.956+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c195370 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.956+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f733801ac70 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.956+0000 7f73337fe700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f7328038400 0x7f732803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.956+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f733804c2b0 con 0x7f733c104490 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.956+0000 7f7341804700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f7328038400 0x7f732803a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.957+0000 7f7341804700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f7328038400 0x7f732803a8b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f732c006fd0 tx=0x7f732c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:44.686 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:43.961+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7338018740 con 0x7f733c104490 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.159+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f733c062380 con 0x7f733c104490 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.162+0000 7f73337fe700 1 -- 172.21.15.37:0/811744392 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7338018350 con 0x7f733c104490 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f7328038400 msgr2=0x7f732803a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f7328038400 0x7f732803a8b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f732c006fd0 tx=0x7f732c006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 msgr2=0x7f733c19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c19bb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7338011670 tx=0x7f7338011aa0 comp rx=0 tx=0).stop 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 shutdown_connections 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f7328038400 0x7f732803a8b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 --2- 172.21.15.37:0/811744392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f733c104490 0x7f733c19bb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 >> 172.21.15.37:0/811744392 conn(0x7f733c100060 msgr2=0x7f733c190050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 shutdown_connections 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:44.165+0000 7f7344269700 1 -- 172.21.15.37:0/811744392 wait complete. 2024-04-15T22:50:44.687 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling the dashboard module... 2024-04-15T22:50:45.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:45 smithi037 ceph-mon[29191]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:50:45.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:45 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/811744392' entity='client.admin' 2024-04-15T22:50:46.499 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:46 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1796382339' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd9b1e700 1 Processor -- start 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd9b1e700 1 -- start start 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd9b1e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd4105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd9b1e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4106460 con 0x7f0bd4105b10 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd4105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd4105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54244/0 (socket says 172.21.15.37:54244) 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd37fe700 1 -- 172.21.15.37:0/2022390325 learned_addr learned my addr 172.21.15.37:0/2022390325 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.044+0000 7f0bd37fe700 1 -- 172.21.15.37:0/2022390325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd41065a0 con 0x7f0bd4105b10 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd37fe700 1 --2- 172.21.15.37:0/2022390325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd4105f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40098a0 tx=0x7f0bc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a515b581d42bbac server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd27fc700 1 -- 172.21.15.37:0/2022390325 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc4004180 con 0x7f0bd4105b10 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd27fc700 1 -- 172.21.15.37:0/2022390325 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0bc40042e0 con 0x7f0bd4105b10 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd27fc700 1 -- 172.21.15.37:0/2022390325 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc4011af0 con 0x7f0bd4105b10 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/2022390325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 msgr2=0x7f0bd4105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 --2- 172.21.15.37:0/2022390325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd4105f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40098a0 tx=0x7f0bc4009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/2022390325 shutdown_connections 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 --2- 172.21.15.37:0/2022390325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd4105f20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/2022390325 >> 172.21.15.37:0/2022390325 conn(0x7f0bd4101180 msgr2=0x7f0bd4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/2022390325 shutdown_connections 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.045+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/2022390325 wait complete. 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd9b1e700 1 Processor -- start 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd9b1e700 1 -- start start 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd9b1e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd9b1e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd407c180 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54256/0 (socket says 172.21.15.37:54256) 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.046+0000 7f0bd37fe700 1 -- 172.21.15.37:0/1796382339 learned_addr learned my addr 172.21.15.37:0/1796382339 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd37fe700 1 -- 172.21.15.37:0/1796382339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4009550 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd37fe700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd407bc40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4006ba0 tx=0x7f0bc4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc4003b00 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0bc401a8e0 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd407a290 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc40199c0 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.047+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd407a6b0 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.048+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f0bc4019c30 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.048+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd4062380 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.048+0000 7f0bd0ff9700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f0bbc0383f0 0x7f0bbc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.048+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0bc404c5c0 con 0x7f0bd4105b10 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.048+0000 7f0bd2ffd700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f0bbc0383f0 0x7f0bbc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.049+0000 7f0bd2ffd700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f0bbc0383f0 0x7f0bbc03a8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8006fd0 tx=0x7f0bc8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.053+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0bc402d440 con 0x7f0bd4105b10 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:45.311+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0bd41079c0 con 0x7f0bd4105b10 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.165+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f0bc4018c70 con 0x7f0bd4105b10 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.167+0000 7f0bd0ff9700 1 -- 172.21.15.37:0/1796382339 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0bc4023760 con 0x7f0bd4105b10 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f0bbc0383f0 msgr2=0x7f0bbc03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f0bbc0383f0 0x7f0bbc03a8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8006fd0 tx=0x7f0bc8006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 msgr2=0x7f0bd407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd407bc40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4006ba0 tx=0x7f0bc4011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 shutdown_connections 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f0bbc0383f0 0x7f0bbc03a8a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 --2- 172.21.15.37:0/1796382339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0bd4105b10 0x7f0bd407bc40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 >> 172.21.15.37:0/1796382339 conn(0x7f0bd4101180 msgr2=0x7f0bd4101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 shutdown_connections 2024-04-15T22:50:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:46.170+0000 7f0bd9b1e700 1 -- 172.21.15.37:0/1796382339 wait complete. 2024-04-15T22:50:47.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:47 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1796382339' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-15T22:50:47.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:47 smithi037 ceph-mon[29191]: mgrmap e9: smithi037.gtwkck(active, since 26s) 2024-04-15T22:50:48.049 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "active_name": "smithi037.gtwkck", 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d0d994700 1 Processor -- start 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d0d994700 1 -- start start 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d0d994700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d081075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d0d994700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08107b00 con 0x7f1d081071b0 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d06ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d081075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d06ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d081075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54288/0 (socket says 172.21.15.37:54288) 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d06ffd700 1 -- 172.21.15.37:0/1340116371 learned_addr learned my addr 172.21.15.37:0/1340116371 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.173+0000 7f1d06ffd700 1 -- 172.21.15.37:0/1340116371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08107c40 con 0x7f1d081071b0 2024-04-15T22:50:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d06ffd700 1 --2- 172.21.15.37:0/1340116371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d081075c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098a0 tx=0x7f1cf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0be92fb1c776f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d05ffb700 1 -- 172.21.15.37:0/1340116371 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf8004180 con 0x7f1d081071b0 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d05ffb700 1 -- 172.21.15.37:0/1340116371 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1cf80042e0 con 0x7f1d081071b0 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d05ffb700 1 -- 172.21.15.37:0/1340116371 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf8011af0 con 0x7f1d081071b0 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 -- 172.21.15.37:0/1340116371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 msgr2=0x7f1d081075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 --2- 172.21.15.37:0/1340116371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d081075c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098a0 tx=0x7f1cf8009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 -- 172.21.15.37:0/1340116371 shutdown_connections 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 --2- 172.21.15.37:0/1340116371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d081075c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 -- 172.21.15.37:0/1340116371 >> 172.21.15.37:0/1340116371 conn(0x7f1d080ff7e0 msgr2=0x7f1d08101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 -- 172.21.15.37:0/1340116371 shutdown_connections 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.174+0000 7f1d0d994700 1 -- 172.21.15.37:0/1340116371 wait complete. 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.175+0000 7f1d0d994700 1 Processor -- start 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.175+0000 7f1d0d994700 1 -- start start 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d0d994700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d08199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d0d994700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08199db0 con 0x7f1d081071b0 2024-04-15T22:50:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d06ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d08199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d06ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d08199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54296/0 (socket says 172.21.15.37:54296) 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d06ffd700 1 -- 172.21.15.37:0/3671808722 learned_addr learned my addr 172.21.15.37:0/3671808722 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d06ffd700 1 -- 172.21.15.37:0/3671808722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8009550 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d06ffd700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d08199870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006ba0 tx=0x7f1cf8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf8003b00 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1cf801a8e0 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.176+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d08199fb0 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.177+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf80199c0 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.177+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0819a3d0 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.177+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f1cf8019c30 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.178+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d08062380 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.178+0000 7f1d0c992700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 0x7f1cf403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.178+0000 7f1d067fc700 1 -- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 msgr2=0x7f1cf403a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.178+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1cf804d640 con 0x7f1d081071b0 2024-04-15T22:50:48.052 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.178+0000 7f1d067fc700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 0x7f1cf403a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.183+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1cf802d440 con 0x7f1d081071b0 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.379+0000 7f1d067fc700 1 -- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 msgr2=0x7f1cf403a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.379+0000 7f1d067fc700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 0x7f1cf403a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.441+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1d0819d550 con 0x7f1d081071b0 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.442+0000 7f1d0c992700 1 -- 172.21.15.37:0/3671808722 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1cf802b340 con 0x7f1d081071b0 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 msgr2=0x7f1cf403a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 0x7f1cf403a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 msgr2=0x7f1d08199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d08199870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8006ba0 tx=0x7f1cf8011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 shutdown_connections 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f1cf4038440 0x7f1cf403a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 --2- 172.21.15.37:0/3671808722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d081071b0 0x7f1d08199870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 >> 172.21.15.37:0/3671808722 conn(0x7f1d080ff7e0 msgr2=0x7f1d081005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 shutdown_connections 2024-04-15T22:50:48.053 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:47.444+0000 7f1d0d994700 1 -- 172.21.15.37:0/3671808722 wait complete. 2024-04-15T22:50:48.054 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for the mgr to restart... 2024-04-15T22:50:48.054 INFO:teuthology.orchestra.run.smithi037.stdout:Waiting for mgr epoch 9... 2024-04-15T22:50:48.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:48 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3671808722' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T22:50:53.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: Active manager daemon smithi037.gtwkck restarted 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: Activating manager daemon smithi037.gtwkck 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: osdmap e3: 0 total, 0 up, 0 in 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: mgrmap e10: smithi037.gtwkck(active, starting, since 0.00379748s) 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr metadata", "who": "smithi037.gtwkck", "id": "smithi037.gtwkck"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: Manager daemon smithi037.gtwkck is now available 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:50:53.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/trash_purge_schedule"}]: dispatch 2024-04-15T22:50:54.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout { 2024-04-15T22:50:54.860 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout } 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f997e4700 1 Processor -- start 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f997e4700 1 -- start start 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f997e4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94108a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f997e4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940744d0 con 0x7f6f94106650 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f92ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94108a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f92ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94108a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54306/0 (socket says 172.21.15.37:54306) 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.532+0000 7f6f92ffd700 1 -- 172.21.15.37:0/2109007790 learned_addr learned my addr 172.21.15.37:0/2109007790 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.533+0000 7f6f92ffd700 1 -- 172.21.15.37:0/2109007790 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94074610 con 0x7f6f94106650 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.533+0000 7f6f92ffd700 1 --2- 172.21.15.37:0/2109007790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94108a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6f840098a0 tx=0x7f6f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521d6d134b53dcf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.533+0000 7f6f91ffb700 1 -- 172.21.15.37:0/2109007790 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f84004180 con 0x7f6f94106650 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.533+0000 7f6f91ffb700 1 -- 172.21.15.37:0/2109007790 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6f840042e0 con 0x7f6f94106650 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.533+0000 7f6f91ffb700 1 -- 172.21.15.37:0/2109007790 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f84011af0 con 0x7f6f94106650 2024-04-15T22:50:54.861 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 -- 172.21.15.37:0/2109007790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 msgr2=0x7f6f94108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 --2- 172.21.15.37:0/2109007790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94108a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6f840098a0 tx=0x7f6f84009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 -- 172.21.15.37:0/2109007790 shutdown_connections 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 --2- 172.21.15.37:0/2109007790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94108a30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 -- 172.21.15.37:0/2109007790 >> 172.21.15.37:0/2109007790 conn(0x7f6f94100070 msgr2=0x7f6f94102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 -- 172.21.15.37:0/2109007790 shutdown_connections 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 -- 172.21.15.37:0/2109007790 wait complete. 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.534+0000 7f6f997e4700 1 Processor -- start 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f997e4700 1 -- start start 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f997e4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94105fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f997e4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f941064f0 con 0x7f6f94106650 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f92ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94105fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f92ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94105fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54314/0 (socket says 172.21.15.37:54314) 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f92ffd700 1 -- 172.21.15.37:0/3784813025 learned_addr learned my addr 172.21.15.37:0/3784813025 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f92ffd700 1 -- 172.21.15.37:0/3784813025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f84009550 con 0x7f6f94106650 2024-04-15T22:50:54.862 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.535+0000 7f6f92ffd700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94105fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f84006ba0 tx=0x7f6f84011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.536+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f84003b00 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.536+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6f8401a8e0 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.536+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f94104660 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.536+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f84019990 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.536+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f94104ae0 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.537+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f6f84019c50 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.537+0000 7f6f7bfff700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.537+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 --> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f7c03afb0 con 0x7f6f7c0383f0 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.537+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6f8404c7d0 con 0x7f6f94106650 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.537+0000 7f6f927fc700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 msgr2=0x7f6f7c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.537+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.738+0000 7f6f927fc700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 msgr2=0x7f6f7c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:48.738+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:49.139+0000 7f6f927fc700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 msgr2=0x7f6f7c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:54.863 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:49.139+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:49.940+0000 7f6f927fc700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 msgr2=0x7f6f7c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:49.940+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:51.541+0000 7f6f927fc700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 msgr2=0x7f6f7c03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/85756762 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:51.541+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:53.074+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mgrmap(e 10) v1 ==== 44864+0+0 (secure 0 0 0) 0x7f6f8401a430 con 0x7f6f94106650 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:53.074+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 msgr2=0x7f6f7c03a8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:53.074+0000 7f6f7bfff700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.075+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f6f8404e440 con 0x7f6f94106650 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.075+0000 7f6f7bfff700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6f7c03be50 0x7f6f7c03e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.075+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f7c03afb0 con 0x7f6f7c03be50 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.075+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6f7c03be50 0x7f6f7c03e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.076+0000 7f6f927fc700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6f7c03be50 0x7f6f7c03e230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6f88003a10 tx=0x7f6f880092b0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.077+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f6f7c03afb0 con 0x7f6f7c03be50 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.088+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6f94105b50 con 0x7f6f7c03be50 2024-04-15T22:50:54.864 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.088+0000 7f6f7bfff700 1 -- 172.21.15.37:0/3784813025 <== mgr.14162 v2:172.21.15.37:6800/712008483 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f6f94105b50 con 0x7f6f7c03be50 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.088+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6f7c03be50 msgr2=0x7f6f7c03e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.088+0000 7f6f997e4700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6f7c03be50 0x7f6f7c03e230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6f88003a10 tx=0x7f6f880092b0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.088+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 msgr2=0x7f6f94105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.088+0000 7f6f997e4700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94105fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f84006ba0 tx=0x7f6f84011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 shutdown_connections 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6f7c03be50 0x7f6f7c03e230 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f94106650 0x7f6f94105fb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 --2- 172.21.15.37:0/3784813025 >> [v2:172.21.15.37:6800/85756762,v1:172.21.15.37:6801/85756762] conn(0x7f6f7c0383f0 0x7f6f7c03a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 >> 172.21.15.37:0/3784813025 conn(0x7f6f94100070 msgr2=0x7f6f94101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 shutdown_connections 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:54.089+0000 7f6f997e4700 1 -- 172.21.15.37:0/3784813025 wait complete. 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:mgr epoch 9 is available 2024-04-15T22:50:54.865 INFO:teuthology.orchestra.run.smithi037.stdout:Generating a dashboard self-signed certificate... 2024-04-15T22:50:55.237 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:55 smithi037 ceph-mon[29191]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T22:50:55.237 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:55 smithi037 ceph-mon[29191]: mgrmap e11: smithi037.gtwkck(active, since 1.00544s) 2024-04-15T22:50:55.237 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:55 smithi037 ceph-mon[29191]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T22:50:55.237 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:55 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:56.115 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78ab25f700 1 Processor -- start 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78ab25f700 1 -- start start 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78ab25f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a40791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78ab25f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4079720 con 0x7f78a407ace0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78a8ffb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a40791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78a8ffb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a40791e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47236/0 (socket says 172.21.15.37:47236) 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78a8ffb700 1 -- 172.21.15.37:0/646059353 learned_addr learned my addr 172.21.15.37:0/646059353 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.271+0000 7f78a8ffb700 1 -- 172.21.15.37:0/646059353 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a4079860 con 0x7f78a407ace0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.272+0000 7f78a8ffb700 1 --2- 172.21.15.37:0/646059353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a40791e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f78a00098a0 tx=0x7f78a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67d7f75b883a69d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.272+0000 7f789b7fe700 1 -- 172.21.15.37:0/646059353 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78a0004180 con 0x7f78a407ace0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.272+0000 7f789b7fe700 1 -- 172.21.15.37:0/646059353 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f78a00042e0 con 0x7f78a407ace0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.272+0000 7f789b7fe700 1 -- 172.21.15.37:0/646059353 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78a0011af0 con 0x7f78a407ace0 2024-04-15T22:50:56.116 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.272+0000 7f78ab25f700 1 -- 172.21.15.37:0/646059353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 msgr2=0x7f78a40791e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.272+0000 7f78ab25f700 1 --2- 172.21.15.37:0/646059353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a40791e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f78a00098a0 tx=0x7f78a0009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 -- 172.21.15.37:0/646059353 shutdown_connections 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 --2- 172.21.15.37:0/646059353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a40791e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 -- 172.21.15.37:0/646059353 >> 172.21.15.37:0/646059353 conn(0x7f78a4101160 msgr2=0x7f78a4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 -- 172.21.15.37:0/646059353 shutdown_connections 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 -- 172.21.15.37:0/646059353 wait complete. 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 Processor -- start 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 -- start start 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.273+0000 7f78ab25f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a419c000 con 0x7f78a407ace0 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f78a8ffb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f78a8ffb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47244/0 (socket says 172.21.15.37:47244) 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f78a8ffb700 1 -- 172.21.15.37:0/3958280643 learned_addr learned my addr 172.21.15.37:0/3958280643 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:56.117 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f78a8ffb700 1 -- 172.21.15.37:0/3958280643 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0009550 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f78a8ffb700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a419bac0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f78a0000c00 tx=0x7f78a0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78a0003a70 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f78a0003bd0 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a419c200 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.274+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78a0023700 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.275+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a419c620 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.275+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a410be40 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.275+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f78a00239c0 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.276+0000 7f7899ffb700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f788c038320 0x7f788c03a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.276+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f78a004c090 con 0x7f78a407ace0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.276+0000 7f789bfff700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f788c038320 0x7f788c03a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.276+0000 7f789bfff700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f788c038320 0x7f788c03a7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7894006fd0 tx=0x7f7894006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.281+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f78a0010de0 con 0x7f78a407ace0 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.377+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f78a002ad20 con 0x7f78a407ace0 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.491+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f78a419f480 con 0x7f788c038320 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.549+0000 7f7899ffb700 1 -- 172.21.15.37:0/3958280643 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f78a419f480 con 0x7f788c038320 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f788c038320 msgr2=0x7f788c03a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f788c038320 0x7f788c03a7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7894006fd0 tx=0x7f7894006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 msgr2=0x7f78a419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a419bac0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f78a0000c00 tx=0x7f78a0011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 shutdown_connections 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f788c038320 0x7f788c03a7d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 --2- 172.21.15.37:0/3958280643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78a407ace0 0x7f78a419bac0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 >> 172.21.15.37:0/3958280643 conn(0x7f78a4101160 msgr2=0x7f78a4101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 shutdown_connections 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:55.551+0000 7f78ab25f700 1 -- 172.21.15.37:0/3958280643 wait complete. 2024-04-15T22:50:56.119 INFO:teuthology.orchestra.run.smithi037.stdout:Creating initial admin user... 2024-04-15T22:50:56.238 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:54] ENGINE Bus STARTING 2024-04-15T22:50:56.238 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:54] ENGINE Serving on https://172.21.15.37:7150 2024-04-15T22:50:56.238 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:54] ENGINE Serving on http://172.21.15.37:8765 2024-04-15T22:50:56.238 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: [15/Apr/2024:22:50:54] ENGINE Bus STARTED 2024-04-15T22:50:56.238 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: mgrmap e12: smithi037.gtwkck(active, since 2s) 2024-04-15T22:50:56.239 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:56.239 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:56.239 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:56 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:57.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:57 smithi037 ceph-mon[29191]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:57.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:57 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:50:57.725 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$y6qYOoEinH6Mm/5iQ4gjveiEA5vmuwam196VFrzRyn931yHdCh6au", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713221457, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.543+0000 7f761e145700 1 Processor -- start 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.543+0000 7f761e145700 1 -- start start 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.543+0000 7f761e145700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f76181063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f761e145700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7618106910 con 0x7f7618105fc0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f76181063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f76181063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47258/0 (socket says 172.21.15.37:47258) 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76177fe700 1 -- 172.21.15.37:0/4104748652 learned_addr learned my addr 172.21.15.37:0/4104748652 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76177fe700 1 -- 172.21.15.37:0/4104748652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7618106a50 con 0x7f7618105fc0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76177fe700 1 --2- 172.21.15.37:0/4104748652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f76181063d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76080098a0 tx=0x7f7608009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353cc93d7e8d5e2f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76167fc700 1 -- 172.21.15.37:0/4104748652 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7608004180 con 0x7f7618105fc0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76167fc700 1 -- 172.21.15.37:0/4104748652 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f76080042e0 con 0x7f7618105fc0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.544+0000 7f76167fc700 1 -- 172.21.15.37:0/4104748652 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7608011af0 con 0x7f7618105fc0 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 -- 172.21.15.37:0/4104748652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 msgr2=0x7f76181063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:57.726 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 --2- 172.21.15.37:0/4104748652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f76181063d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76080098a0 tx=0x7f7608009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 -- 172.21.15.37:0/4104748652 shutdown_connections 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 --2- 172.21.15.37:0/4104748652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f76181063d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 -- 172.21.15.37:0/4104748652 >> 172.21.15.37:0/4104748652 conn(0x7f76180ff7e0 msgr2=0x7f7618101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 -- 172.21.15.37:0/4104748652 shutdown_connections 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.545+0000 7f761e145700 1 -- 172.21.15.37:0/4104748652 wait complete. 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.546+0000 7f761e145700 1 Processor -- start 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.546+0000 7f761e145700 1 -- start start 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.546+0000 7f761e145700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f7618104770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.546+0000 7f761e145700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761819c5c0 con 0x7f7618105fc0 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f76177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f7618104770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f76177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f7618104770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47262/0 (socket says 172.21.15.37:47262) 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f76177fe700 1 -- 172.21.15.37:0/2304442756 learned_addr learned my addr 172.21.15.37:0/2304442756 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:57.727 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f76177fe700 1 -- 172.21.15.37:0/2304442756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7608009550 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f76177fe700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f7618104770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7608006ba0 tx=0x7f7608011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7608003b00 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f760801a8e0 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761819c700 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.547+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7608019940 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.548+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761819cb20 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.548+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f7608019bb0 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.548+0000 7f7614ff9700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7600038480 0x7f760003a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.548+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f760801a630 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.548+0000 7f7616ffd700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7600038480 0x7f760003a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.549+0000 7f7616ffd700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7600038480 0x7f760003a930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f760c006fd0 tx=0x7f760c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.549+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7618062380 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.554+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f760802d440 con 0x7f7618105fc0 2024-04-15T22:50:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:56.773+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7618105920 con 0x7f7600038480 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.020+0000 7f7614ff9700 1 -- 172.21.15.37:0/2304442756 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7618105920 con 0x7f7600038480 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7600038480 msgr2=0x7f760003a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7600038480 0x7f760003a930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f760c006fd0 tx=0x7f760c006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 msgr2=0x7f7618104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f7618104770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7608006ba0 tx=0x7f7608011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 shutdown_connections 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7600038480 0x7f760003a930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 --2- 172.21.15.37:0/2304442756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7618105fc0 0x7f7618104770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 >> 172.21.15.37:0/2304442756 conn(0x7f76180ff7e0 msgr2=0x7f7618100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 shutdown_connections 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:57.022+0000 7f761e145700 1 -- 172.21.15.37:0/2304442756 wait complete. 2024-04-15T22:50:57.729 INFO:teuthology.orchestra.run.smithi037.stdout:Fetching dashboard port number... 2024-04-15T22:50:58.456 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:58 smithi037 ceph-mon[29191]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:50:59.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:50:59 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1577971039' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-15T22:50:59.349 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stdout 8443 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.649+0000 7f40d5e13700 1 Processor -- start 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.649+0000 7f40d5e13700 1 -- start start 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.650+0000 7f40d5e13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0107520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.650+0000 7f40d5e13700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d0107a60 con 0x7f40d0107110 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.650+0000 7f40cf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.650+0000 7f40cf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0107520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47278/0 (socket says 172.21.15.37:47278) 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.650+0000 7f40cf7fe700 1 -- 172.21.15.37:0/3828341944 learned_addr learned my addr 172.21.15.37:0/3828341944 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.650+0000 7f40cf7fe700 1 -- 172.21.15.37:0/3828341944 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0107ba0 con 0x7f40d0107110 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40cf7fe700 1 --2- 172.21.15.37:0/3828341944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0107520 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f40c00098a0 tx=0x7f40c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb500322b6fbb73 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40ceffd700 1 -- 172.21.15.37:0/3828341944 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c0004180 con 0x7f40d0107110 2024-04-15T22:50:59.356 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40ceffd700 1 -- 172.21.15.37:0/3828341944 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f40c00042e0 con 0x7f40d0107110 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40ceffd700 1 -- 172.21.15.37:0/3828341944 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c0011af0 con 0x7f40d0107110 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40d5e13700 1 -- 172.21.15.37:0/3828341944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 msgr2=0x7f40d0107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40d5e13700 1 --2- 172.21.15.37:0/3828341944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0107520 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f40c00098a0 tx=0x7f40c0009bb0 comp rx=0 tx=0).stop 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40d5e13700 1 -- 172.21.15.37:0/3828341944 shutdown_connections 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40d5e13700 1 --2- 172.21.15.37:0/3828341944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0107520 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40d5e13700 1 -- 172.21.15.37:0/3828341944 >> 172.21.15.37:0/3828341944 conn(0x7f40d00ff7e0 msgr2=0x7f40d0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.651+0000 7f40d5e13700 1 -- 172.21.15.37:0/3828341944 shutdown_connections 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.652+0000 7f40d5e13700 1 -- 172.21.15.37:0/3828341944 wait complete. 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.652+0000 7f40d5e13700 1 Processor -- start 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.652+0000 7f40d5e13700 1 -- start start 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40d5e13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40d5e13700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d0199db0 con 0x7f40d0107110 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47280/0 (socket says 172.21.15.37:47280) 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cf7fe700 1 -- 172.21.15.37:0/1577971039 learned_addr learned my addr 172.21.15.37:0/1577971039 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:50:59.357 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cf7fe700 1 -- 172.21.15.37:0/1577971039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0009550 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cf7fe700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0199870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40c0006ba0 tx=0x7f40c0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c0003b00 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d0199fb0 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f40c001a8e0 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c0019990 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.653+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d019a3d0 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.654+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f40c0019c50 con 0x7f40d0107110 2024-04-15T22:50:59.358 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.654+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d0062380 con 0x7f40d0107110 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.655+0000 7f40cd7fa700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f40bc0383f0 0x7f40bc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.655+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f40c004c700 con 0x7f40d0107110 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.655+0000 7f40c6ffd700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f40bc0383f0 0x7f40bc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.655+0000 7f40c6ffd700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f40bc0383f0 0x7f40bc03a8a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006fd0 tx=0x7f40b8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.660+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f40c002b700 con 0x7f40d0107110 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.876+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f40d019d2c0 con 0x7f40d0107110 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.876+0000 7f40cd7fa700 1 -- 172.21.15.37:0/1577971039 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f40c002b340 con 0x7f40d0107110 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f40bc0383f0 msgr2=0x7f40bc03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f40bc0383f0 0x7f40bc03a8a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006fd0 tx=0x7f40b8006e40 comp rx=0 tx=0).stop 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 msgr2=0x7f40d0199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0199870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40c0006ba0 tx=0x7f40c0011b50 comp rx=0 tx=0).stop 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 shutdown_connections 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f40bc0383f0 0x7f40bc03a8a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:59.359 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 --2- 172.21.15.37:0/1577971039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40d0107110 0x7f40d0199870 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:50:59.360 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 >> 172.21.15.37:0/1577971039 conn(0x7f40d00ff7e0 msgr2=0x7f40d0100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:50:59.360 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 shutdown_connections 2024-04-15T22:50:59.360 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:58.880+0000 7f40d5e13700 1 -- 172.21.15.37:0/1577971039 wait complete. 2024-04-15T22:50:59.362 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T22:50:59.362 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout disabled 2024-04-15T22:50:59.369 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T22:50:59.369 INFO:teuthology.orchestra.run.smithi037.stdout:systemctl: stdout inactive 2024-04-15T22:50:59.369 INFO:teuthology.orchestra.run.smithi037.stdout:firewalld.service is not enabled 2024-04-15T22:50:59.369 INFO:teuthology.orchestra.run.smithi037.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-15T22:50:59.371 INFO:teuthology.orchestra.run.smithi037.stdout:Ceph Dashboard is now available at: 2024-04-15T22:50:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:50:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: URL: https://smithi037.front.sepia.ceph.com:8443/ 2024-04-15T22:50:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: User: admin 2024-04-15T22:50:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: Password: u79tmdy5pe 2024-04-15T22:50:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:50:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:Saving cluster configuration to /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config directory 2024-04-15T22:50:59.372 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling autotune for osd_memory_target 2024-04-15T22:51:00.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:00 smithi037 ceph-mon[29191]: mgrmap e13: smithi037.gtwkck(active, since 6s) 2024-04-15T22:51:00.641 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.846+0000 7fb8c53b6700 1 Processor -- start 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.846+0000 7fb8c53b6700 1 -- start start 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.846+0000 7fb8c53b6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 0x7fb8c010aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.846+0000 7fb8c53b6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01064c0 con 0x7fb8c0079cf0 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 0x7fb8c010aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 0x7fb8c010aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49220/0 (socket says 172.21.15.37:49220) 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8beffd700 1 -- 172.21.15.37:0/2843035704 learned_addr learned my addr 172.21.15.37:0/2843035704 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8beffd700 1 -- 172.21.15.37:0/2843035704 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c010b010 con 0x7fb8c0079cf0 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8beffd700 1 --2- 172.21.15.37:0/2843035704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 0x7fb8c010aad0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00098a0 tx=0x7fb8b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a82bcfff082f507 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8bdffb700 1 -- 172.21.15.37:0/2843035704 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b0004180 con 0x7fb8c0079cf0 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8bdffb700 1 -- 172.21.15.37:0/2843035704 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb8b00042e0 con 0x7fb8c0079cf0 2024-04-15T22:51:00.642 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.847+0000 7fb8bdffb700 1 -- 172.21.15.37:0/2843035704 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b0011af0 con 0x7fb8c0079cf0 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 -- 172.21.15.37:0/2843035704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 msgr2=0x7fb8c010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 --2- 172.21.15.37:0/2843035704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 0x7fb8c010aad0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00098a0 tx=0x7fb8b0009bb0 comp rx=0 tx=0).stop 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 -- 172.21.15.37:0/2843035704 shutdown_connections 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 --2- 172.21.15.37:0/2843035704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c0079cf0 0x7fb8c010aad0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 -- 172.21.15.37:0/2843035704 >> 172.21.15.37:0/2843035704 conn(0x7fb8c0100060 msgr2=0x7fb8c01024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 -- 172.21.15.37:0/2843035704 shutdown_connections 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.848+0000 7fb8c53b6700 1 -- 172.21.15.37:0/2843035704 wait complete. 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8c53b6700 1 Processor -- start 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8c53b6700 1 -- start start 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8c53b6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 0x7fb8c019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8c53b6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c019c440 con 0x7fb8c019baf0 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 0x7fb8c019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8beffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 0x7fb8c019bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49236/0 (socket says 172.21.15.37:49236) 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8beffd700 1 -- 172.21.15.37:0/1040825513 learned_addr learned my addr 172.21.15.37:0/1040825513 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:00.643 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.849+0000 7fb8beffd700 1 -- 172.21.15.37:0/1040825513 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b0009550 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.850+0000 7fb8beffd700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 0x7fb8c019bf00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0011670 tx=0x7fb8b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.850+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b00039c0 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.850+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb8b0003b20 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.850+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c019c640 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.850+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b00247f0 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.850+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c019f2a0 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.851+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fb8b0024a60 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.851+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c0195b60 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.851+0000 7fb8b7fff700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fb8ac0384f0 0x7fb8ac03a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.851+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb8b004c6c0 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.851+0000 7fb8be7fc700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fb8ac0384f0 0x7fb8ac03a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.851+0000 7fb8be7fc700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fb8ac0384f0 0x7fb8ac03a9a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:50:59.856+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb8b0022720 con 0x7fb8c019baf0 2024-04-15T22:51:00.644 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.057+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb8c0079cf0 con 0x7fb8c019baf0 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.057+0000 7fb8b7fff700 1 -- 172.21.15.37:0/1040825513 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb8b002c350 con 0x7fb8c019baf0 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.059+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fb8ac0384f0 msgr2=0x7fb8ac03a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.059+0000 7fb8c53b6700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fb8ac0384f0 0x7fb8ac03a9a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.059+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 msgr2=0x7fb8c019bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.059+0000 7fb8c53b6700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 0x7fb8c019bf00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0011670 tx=0x7fb8b0011aa0 comp rx=0 tx=0).stop 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.060+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 shutdown_connections 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.060+0000 7fb8c53b6700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fb8ac0384f0 0x7fb8ac03a9a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.060+0000 7fb8c53b6700 1 --2- 172.21.15.37:0/1040825513 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8c019baf0 0x7fb8c019bf00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.060+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 >> 172.21.15.37:0/1040825513 conn(0x7fb8c0100060 msgr2=0x7fb8c0101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.060+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 shutdown_connections 2024-04-15T22:51:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:00.060+0000 7fb8c53b6700 1 -- 172.21.15.37:0/1040825513 wait complete. 2024-04-15T22:51:01.940 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f6d88700 1 Processor -- start 2024-04-15T22:51:01.940 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f6d88700 1 -- start start 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f6d88700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f6d88700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0105860 con 0x7f19f0104f10 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f4b24700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f4b24700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49248/0 (socket says 172.21.15.37:49248) 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f4b24700 1 -- 172.21.15.37:0/1420992659 learned_addr learned my addr 172.21.15.37:0/1420992659 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.029+0000 7f19f4b24700 1 -- 172.21.15.37:0/1420992659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f01059a0 con 0x7f19f0104f10 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f4b24700 1 --2- 172.21.15.37:0/1420992659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0105320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19e40098a0 tx=0x7f19e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ddc4e923decf77 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19ef7fe700 1 -- 172.21.15.37:0/1420992659 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19e4004180 con 0x7f19f0104f10 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19ef7fe700 1 -- 172.21.15.37:0/1420992659 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f19e40042e0 con 0x7f19f0104f10 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19ef7fe700 1 -- 172.21.15.37:0/1420992659 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19e4011af0 con 0x7f19f0104f10 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f6d88700 1 -- 172.21.15.37:0/1420992659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 msgr2=0x7f19f0105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f6d88700 1 --2- 172.21.15.37:0/1420992659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0105320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19e40098a0 tx=0x7f19e4009bb0 comp rx=0 tx=0).stop 2024-04-15T22:51:01.941 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f6d88700 1 -- 172.21.15.37:0/1420992659 shutdown_connections 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f6d88700 1 --2- 172.21.15.37:0/1420992659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0105320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f6d88700 1 -- 172.21.15.37:0/1420992659 >> 172.21.15.37:0/1420992659 conn(0x7f19f00ff7e0 msgr2=0x7f19f0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.030+0000 7f19f6d88700 1 -- 172.21.15.37:0/1420992659 shutdown_connections 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f6d88700 1 -- 172.21.15.37:0/1420992659 wait complete. 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f6d88700 1 Processor -- start 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f6d88700 1 -- start start 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f6d88700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f6d88700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0198cd0 con 0x7f19f0104f10 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f4b24700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.031+0000 7f19f4b24700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49258/0 (socket says 172.21.15.37:49258) 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19f4b24700 1 -- 172.21.15.37:0/2973016651 learned_addr learned my addr 172.21.15.37:0/2973016651 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19f4b24700 1 -- 172.21.15.37:0/2973016651 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4009550 con 0x7f19f0104f10 2024-04-15T22:51:01.942 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19f4b24700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0198790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f19e4006ba0 tx=0x7f19e4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19e4003b00 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f19e401a8e0 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f0198ed0 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19e4019990 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.032+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f01992f0 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.033+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f19e4019c00 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.033+0000 7f19edffb700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f19e00384e0 0x7f19e003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.033+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f19e404c7b0 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.033+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0062380 con 0x7f19f0104f10 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.033+0000 7f19effff700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f19e00384e0 0x7f19e003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:01.943 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.034+0000 7f19effff700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f19e00384e0 0x7f19e003a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19dc006fd0 tx=0x7f19dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.038+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f19e402d440 con 0x7f19f0104f10 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.349+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f19f0106930 con 0x7f19f0104f10 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.352+0000 7f19edffb700 1 -- 172.21.15.37:0/2973016651 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f19e402b340 con 0x7f19f0104f10 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f19e00384e0 msgr2=0x7f19e003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f19e00384e0 0x7f19e003a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19dc006fd0 tx=0x7f19dc006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 msgr2=0x7f19f0198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0198790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f19e4006ba0 tx=0x7f19e4011b50 comp rx=0 tx=0).stop 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 shutdown_connections 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f19e00384e0 0x7f19e003a990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 --2- 172.21.15.37:0/2973016651 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19f0104f10 0x7f19f0198790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 >> 172.21.15.37:0/2973016651 conn(0x7f19f00ff7e0 msgr2=0x7f19f0100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 shutdown_connections 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr 2024-04-15T22:51:01.354+0000 7f19f6d88700 1 -- 172.21.15.37:0/2973016651 wait complete. 2024-04-15T22:51:01.944 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout:Or, if you are only running a single cluster on this host: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: ceph telemetry on 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout:For more information see: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-15T22:51:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:01.946 INFO:teuthology.orchestra.run.smithi037.stdout:Bootstrap complete. 2024-04-15T22:51:01.976 INFO:tasks.cephadm:Fetching config... 2024-04-15T22:51:01.977 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:51:01.977 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-15T22:51:02.035 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-15T22:51:02.035 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:51:02.035 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-15T22:51:02.091 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-15T22:51:02.091 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:51:02.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/keyring of=/dev/stdout 2024-04-15T22:51:02.160 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-15T22:51:02.160 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:51:02.160 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-15T22:51:02.216 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-15T22:51:02.216 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlmzEVRQ59AGuVSQJDhl/p2+9m1H/5sowcd707oL28VLg0B43YGXR9QyM5V9VryBxROMf7eD6kOZvrMs1NGdH3pJ4YZHl314FlxZuFnZDsKGslZ1Cb+4WqZrmlkEs9SSz6ItlI9pDD1J4kRztHXGGJfRG/n+kczP9oMrFuMkGkjyEFgvGeYD+REz4L9ptsX/A1//H9JBs1TRQNQMFhbiIB7m9zDBUU4EkDad0TaQ9EkFcVlm3lbdNELnCMyE1ZbUync8cyffCuQA5Jr3wRMrFVEJVhTjAX2XCCFoGUAiMfK/6lUcdaHewpvAxXffdIokl6jKBhxhV4X9DUHmPuubCTdE1VhjvRwboiU8a5PCdh3Jzc6ai5/TE43g2ulyPSi3+5bwt++HSNwGbPmL5BU5pr3zJKgaPG9+ZwUsrUzIDJWe7omktLanE4tnUBbMTHvE8K3oeQKsbzttc/RtTDsZsJnSZFg/Eqe5Na3Ps4ZzPMNsvfhtHaLgAAHNHwfcCTs1c= ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T22:51:02.312 INFO:teuthology.orchestra.run.smithi037.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlmzEVRQ59AGuVSQJDhl/p2+9m1H/5sowcd707oL28VLg0B43YGXR9QyM5V9VryBxROMf7eD6kOZvrMs1NGdH3pJ4YZHl314FlxZuFnZDsKGslZ1Cb+4WqZrmlkEs9SSz6ItlI9pDD1J4kRztHXGGJfRG/n+kczP9oMrFuMkGkjyEFgvGeYD+REz4L9ptsX/A1//H9JBs1TRQNQMFhbiIB7m9zDBUU4EkDad0TaQ9EkFcVlm3lbdNELnCMyE1ZbUync8cyffCuQA5Jr3wRMrFVEJVhTjAX2XCCFoGUAiMfK/6lUcdaHewpvAxXffdIokl6jKBhxhV4X9DUHmPuubCTdE1VhjvRwboiU8a5PCdh3Jzc6ai5/TE43g2ulyPSi3+5bwt++HSNwGbPmL5BU5pr3zJKgaPG9+ZwUsrUzIDJWe7omktLanE4tnUBbMTHvE8K3oeQKsbzttc/RtTDsZsJnSZFg/Eqe5Na3Ps4ZzPMNsvfhtHaLgAAHNHwfcCTs1c= ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:51:02.327 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlmzEVRQ59AGuVSQJDhl/p2+9m1H/5sowcd707oL28VLg0B43YGXR9QyM5V9VryBxROMf7eD6kOZvrMs1NGdH3pJ4YZHl314FlxZuFnZDsKGslZ1Cb+4WqZrmlkEs9SSz6ItlI9pDD1J4kRztHXGGJfRG/n+kczP9oMrFuMkGkjyEFgvGeYD+REz4L9ptsX/A1//H9JBs1TRQNQMFhbiIB7m9zDBUU4EkDad0TaQ9EkFcVlm3lbdNELnCMyE1ZbUync8cyffCuQA5Jr3wRMrFVEJVhTjAX2XCCFoGUAiMfK/6lUcdaHewpvAxXffdIokl6jKBhxhV4X9DUHmPuubCTdE1VhjvRwboiU8a5PCdh3Jzc6ai5/TE43g2ulyPSi3+5bwt++HSNwGbPmL5BU5pr3zJKgaPG9+ZwUsrUzIDJWe7omktLanE4tnUBbMTHvE8K3oeQKsbzttc/RtTDsZsJnSZFg/Eqe5Na3Ps4ZzPMNsvfhtHaLgAAHNHwfcCTs1c= ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T22:51:02.380 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlmzEVRQ59AGuVSQJDhl/p2+9m1H/5sowcd707oL28VLg0B43YGXR9QyM5V9VryBxROMf7eD6kOZvrMs1NGdH3pJ4YZHl314FlxZuFnZDsKGslZ1Cb+4WqZrmlkEs9SSz6ItlI9pDD1J4kRztHXGGJfRG/n+kczP9oMrFuMkGkjyEFgvGeYD+REz4L9ptsX/A1//H9JBs1TRQNQMFhbiIB7m9zDBUU4EkDad0TaQ9EkFcVlm3lbdNELnCMyE1ZbUync8cyffCuQA5Jr3wRMrFVEJVhTjAX2XCCFoGUAiMfK/6lUcdaHewpvAxXffdIokl6jKBhxhV4X9DUHmPuubCTdE1VhjvRwboiU8a5PCdh3Jzc6ai5/TE43g2ulyPSi3+5bwt++HSNwGbPmL5BU5pr3zJKgaPG9+ZwUsrUzIDJWe7omktLanE4tnUBbMTHvE8K3oeQKsbzttc/RtTDsZsJnSZFg/Eqe5Na3Ps4ZzPMNsvfhtHaLgAAHNHwfcCTs1c= ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:51:02.396 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-15T22:51:02.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:02 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2973016651' entity='client.admin' 2024-04-15T22:51:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.270+0000 7f3c86279700 1 -- 172.21.15.37:0/624364856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 msgr2=0x7f3c800ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.270+0000 7f3c86279700 1 --2- 172.21.15.37:0/624364856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c800ff130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009910 tx=0x7f3c70009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 -- 172.21.15.37:0/624364856 shutdown_connections 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 --2- 172.21.15.37:0/624364856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c800ff130 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 -- 172.21.15.37:0/624364856 >> 172.21.15.37:0/624364856 conn(0x7f3c800fa2b0 msgr2=0x7f3c800fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 -- 172.21.15.37:0/624364856 shutdown_connections 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 -- 172.21.15.37:0/624364856 wait complete. 2024-04-15T22:51:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 Processor -- start 2024-04-15T22:51:05.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.271+0000 7f3c86279700 1 -- start start 2024-04-15T22:51:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c86279700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c801971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c86279700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c801976e0 con 0x7f3c800fed20 2024-04-15T22:51:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c801971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c801971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49278/0 (socket says 172.21.15.37:49278) 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7ffff700 1 -- 172.21.15.37:0/826110390 learned_addr learned my addr 172.21.15.37:0/826110390 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7ffff700 1 -- 172.21.15.37:0/826110390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700095f0 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7ffff700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c801971a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c70005b40 tx=0x7f3c70004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7001d070 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3c7000be10 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7000f460 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c801978e0 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.272+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c80197d80 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.273+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f3c7000f5c0 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.273+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c80190e10 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.273+0000 7f3c7d7fa700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3c680384e0 0x7f3c6803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.273+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3c7004c4d0 con 0x7f3c800fed20 2024-04-15T22:51:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.273+0000 7f3c7f7fe700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3c680384e0 0x7f3c6803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.274+0000 7f3c7f7fe700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3c680384e0 0x7f3c6803a990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c74006fd0 tx=0x7f3c74006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.278+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3c70020550 con 0x7f3c800fed20 2024-04-15T22:51:05.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.488+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f3c8002cc30 con 0x7f3c800fed20 2024-04-15T22:51:05.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.491+0000 7f3c7d7fa700 1 -- 172.21.15.37:0/826110390 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3c70027030 con 0x7f3c800fed20 2024-04-15T22:51:05.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3c680384e0 msgr2=0x7f3c6803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:05.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3c680384e0 0x7f3c6803a990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c74006fd0 tx=0x7f3c74006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:05.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 msgr2=0x7f3c801971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c801971a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c70005b40 tx=0x7f3c70004740 comp rx=0 tx=0).stop 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 shutdown_connections 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3c680384e0 0x7f3c6803a990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 --2- 172.21.15.37:0/826110390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c800fed20 0x7f3c801971a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.493+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 >> 172.21.15.37:0/826110390 conn(0x7f3c800fa2b0 msgr2=0x7f3c800690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.494+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 shutdown_connections 2024-04-15T22:51:05.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:05.494+0000 7f3c86279700 1 -- 172.21.15.37:0/826110390 wait complete. 2024-04-15T22:51:06.173 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-15T22:51:06.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-15T22:51:06.719 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:51:06.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:06 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/826110390' entity='client.admin' 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd/host:smithi037", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi037", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi037", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:51:07.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:07 smithi037 ceph-mon[29191]: Deploying daemon ceph-exporter.smithi037 on smithi037 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.940+0000 7f9ca1c44700 1 -- 172.21.15.37:0/924048815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 msgr2=0x7f9c9c102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.940+0000 7f9ca1c44700 1 --2- 172.21.15.37:0/924048815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c102ca0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009910 tx=0x7f9c8c009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.940+0000 7f9ca1c44700 1 -- 172.21.15.37:0/924048815 shutdown_connections 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.940+0000 7f9ca1c44700 1 --2- 172.21.15.37:0/924048815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c102ca0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.940+0000 7f9ca1c44700 1 -- 172.21.15.37:0/924048815 >> 172.21.15.37:0/924048815 conn(0x7f9c9c0fa2c0 msgr2=0x7f9c9c0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.941+0000 7f9ca1c44700 1 -- 172.21.15.37:0/924048815 shutdown_connections 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.941+0000 7f9ca1c44700 1 -- 172.21.15.37:0/924048815 wait complete. 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.941+0000 7f9ca1c44700 1 Processor -- start 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.941+0000 7f9ca1c44700 1 -- start start 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.941+0000 7f9ca1c44700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.941+0000 7f9ca1c44700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c100760 con 0x7f9c9c1008c0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c9b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c9b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c100220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49300/0 (socket says 172.21.15.37:49300) 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c9b7fe700 1 -- 172.21.15.37:0/532433833 learned_addr learned my addr 172.21.15.37:0/532433833 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c9b7fe700 1 -- 172.21.15.37:0/532433833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c0095f0 con 0x7f9c9c1008c0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c9b7fe700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c100220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c005b40 tx=0x7f9c8c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c8c01d070 con 0x7f9c9c1008c0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c00be10 con 0x7f9c9c1008c0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c0fe870 con 0x7f9c9c1008c0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c8c00f460 con 0x7f9c9c1008c0 2024-04-15T22:51:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.942+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c0fed10 con 0x7f9c9c1008c0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.943+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f9c8c00b960 con 0x7f9c9c1008c0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.943+0000 7f9c98ff9700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9c84038490 0x7f9c8403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.943+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9c8c04c400 con 0x7f9c9c1008c0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.943+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c062380 con 0x7f9c9c1008c0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.943+0000 7f9c9affd700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9c84038490 0x7f9c8403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.944+0000 7f9c9affd700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9c84038490 0x7f9c8403a940 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9c90006fd0 tx=0x7f9c90006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:08.948+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9c8c027070 con 0x7f9c9c1008c0 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.156+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c9c0fefe0 con 0x7f9c84038490 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.159+0000 7f9c98ff9700 1 -- 172.21.15.37:0/532433833 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9c9c0fefe0 con 0x7f9c84038490 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9c84038490 msgr2=0x7f9c8403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9c84038490 0x7f9c8403a940 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9c90006fd0 tx=0x7f9c90006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 msgr2=0x7f9c9c100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c100220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c005b40 tx=0x7f9c8c004740 comp rx=0 tx=0).stop 2024-04-15T22:51:09.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 shutdown_connections 2024-04-15T22:51:09.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9c84038490 0x7f9c8403a940 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:09.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 --2- 172.21.15.37:0/532433833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c9c1008c0 0x7f9c9c100220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:09.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 >> 172.21.15.37:0/532433833 conn(0x7f9c9c0fa2c0 msgr2=0x7f9c9c0faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:09.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 shutdown_connections 2024-04-15T22:51:09.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:09.161+0000 7f9ca1c44700 1 -- 172.21.15.37:0/532433833 wait complete. 2024-04-15T22:51:09.944 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-04-15T22:51:09.944 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:51:09.944 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-04-15T22:51:09.964 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:51:09.964 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:51:10.023 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-04-15T22:51:10.023 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch host add smithi189 2024-04-15T22:51:10.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:10 smithi037 ceph-mon[29191]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:51:10.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:10 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi037", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi037", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-15T22:51:11.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:51:11.821 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:11 smithi037 ceph-mon[29191]: Deploying daemon crash.smithi037 on smithi037 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.069+0000 7f132eb21700 1 -- 172.21.15.37:0/820405554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 msgr2=0x7f1328100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.069+0000 7f132eb21700 1 --2- 172.21.15.37:0/820405554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328100230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f131c009910 tx=0x7f131c009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.070+0000 7f132eb21700 1 -- 172.21.15.37:0/820405554 shutdown_connections 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.070+0000 7f132eb21700 1 --2- 172.21.15.37:0/820405554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328100230 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.070+0000 7f132eb21700 1 -- 172.21.15.37:0/820405554 >> 172.21.15.37:0/820405554 conn(0x7f13280fb3d0 msgr2=0x7f13280fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.070+0000 7f132eb21700 1 -- 172.21.15.37:0/820405554 shutdown_connections 2024-04-15T22:51:12.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.070+0000 7f132eb21700 1 -- 172.21.15.37:0/820405554 wait complete. 2024-04-15T22:51:12.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132eb21700 1 Processor -- start 2024-04-15T22:51:12.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132eb21700 1 -- start start 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132eb21700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132eb21700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13281954b0 con 0x7f13280ffe20 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132c8bd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132c8bd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35122/0 (socket says 172.21.15.37:35122) 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132c8bd700 1 -- 172.21.15.37:0/2491761426 learned_addr learned my addr 172.21.15.37:0/2491761426 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.071+0000 7f132c8bd700 1 -- 172.21.15.37:0/2491761426 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c0095f0 con 0x7f13280ffe20 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.072+0000 7f132c8bd700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328194f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f131c005b40 tx=0x7f131c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.072+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f131c01d070 con 0x7f13280ffe20 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.072+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f131c00be10 con 0x7f13280ffe20 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.072+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13281956b0 con 0x7f13280ffe20 2024-04-15T22:51:12.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.072+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f131c00f460 con 0x7f13280ffe20 2024-04-15T22:51:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.072+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1328195b50 con 0x7f13280ffe20 2024-04-15T22:51:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.073+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f131c00f6d0 con 0x7f13280ffe20 2024-04-15T22:51:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.073+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132818ea80 con 0x7f13280ffe20 2024-04-15T22:51:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.073+0000 7f1325ffb700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f1318038490 0x7f131803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.073+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f131c04c340 con 0x7f13280ffe20 2024-04-15T22:51:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.073+0000 7f1327fff700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f1318038490 0x7f131803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:12.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.074+0000 7f1327fff700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f1318038490 0x7f131803a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1314006fd0 tx=0x7f1314006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:12.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.078+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f131c02c440 con 0x7f13280ffe20 2024-04-15T22:51:12.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:12.285+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) v1 -- 0x7f13281008b0 con 0x7f1318038490 2024-04-15T22:51:14.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:14 smithi037 ceph-mon[29191]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:51:14.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:14 smithi037 ceph-mon[29191]: Deploying cephadm binary to smithi189 2024-04-15T22:51:14.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:14 smithi037 ceph-mon[29191]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:14.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:14 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:15.006 INFO:teuthology.orchestra.run.smithi037.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-04-15T22:51:15.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.007+0000 7f1325ffb700 1 -- 172.21.15.37:0/2491761426 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f13281008b0 con 0x7f1318038490 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f1318038490 msgr2=0x7f131803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f1318038490 0x7f131803a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1314006fd0 tx=0x7f1314006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 msgr2=0x7f1328194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328194f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f131c005b40 tx=0x7f131c004740 comp rx=0 tx=0).stop 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 shutdown_connections 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f1318038490 0x7f131803a940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 --2- 172.21.15.37:0/2491761426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13280ffe20 0x7f1328194f70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 >> 172.21.15.37:0/2491761426 conn(0x7f13280fb3d0 msgr2=0x7f13280fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 shutdown_connections 2024-04-15T22:51:15.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:15.010+0000 7f132eb21700 1 -- 172.21.15.37:0/2491761426 wait complete. 2024-04-15T22:51:15.390 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: Deploying daemon node-exporter.smithi037 on smithi037 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: Added host smithi189 2024-04-15T22:51:15.391 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:15 smithi037 ceph-mon[29191]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:15.679 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-15T22:51:15.903 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 -- 172.21.15.37:0/525464373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 msgr2=0x7f9f7c0ff120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 --2- 172.21.15.37:0/525464373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0ff120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009910 tx=0x7f9f6c009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 -- 172.21.15.37:0/525464373 shutdown_connections 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 --2- 172.21.15.37:0/525464373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0ff120 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 -- 172.21.15.37:0/525464373 >> 172.21.15.37:0/525464373 conn(0x7f9f7c0fa380 msgr2=0x7f9f7c0fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 -- 172.21.15.37:0/525464373 shutdown_connections 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.547+0000 7f9f80ac4700 1 -- 172.21.15.37:0/525464373 wait complete. 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f80ac4700 1 Processor -- start 2024-04-15T22:51:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f80ac4700 1 -- start start 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f80ac4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f80ac4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c197cc0 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f7a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f7a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0693e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35138/0 (socket says 172.21.15.37:35138) 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.548+0000 7f9f7a59c700 1 -- 172.21.15.37:0/754766360 learned_addr learned my addr 172.21.15.37:0/754766360 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f7a59c700 1 -- 172.21.15.37:0/754766360 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0095f0 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f7a59c700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c005b40 tx=0x7f9f6c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f6c01d070 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f6c00be10 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7c197e00 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f6c00f460 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.549+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7c198240 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.550+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f9f6c00f5c0 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.550+0000 7f9f6b7fe700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9f640384e0 0x7f9f6403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.550+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9f6c0227e0 con 0x7f9f7c0fed10 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.550+0000 7f9f79d9b700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9f640384e0 0x7f9f6403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.550+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7c18d970 con 0x7f9f7c0fed10 2024-04-15T22:51:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.551+0000 7f9f79d9b700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9f640384e0 0x7f9f6403a990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9f70006fd0 tx=0x7f9f70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.555+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9f6c027070 con 0x7f9f7c0fed10 2024-04-15T22:51:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.770+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f7c061190 con 0x7f9f640384e0 2024-04-15T22:51:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.771+0000 7f9f6b7fe700 1 -- 172.21.15.37:0/754766360 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f9f7c061190 con 0x7f9f640384e0 2024-04-15T22:51:17.768 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:51:17.768 INFO:teuthology.orchestra.run.smithi037.stdout:[{"addr": "172.21.15.37", "hostname": "smithi037", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-04-15T22:51:17.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9f640384e0 msgr2=0x7f9f6403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:17.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9f640384e0 0x7f9f6403a990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9f70006fd0 tx=0x7f9f70006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:17.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 msgr2=0x7f9f7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:17.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c005b40 tx=0x7f9f6c004740 comp rx=0 tx=0).stop 2024-04-15T22:51:17.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 shutdown_connections 2024-04-15T22:51:17.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9f640384e0 0x7f9f6403a990 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:17.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 --2- 172.21.15.37:0/754766360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7c0fed10 0x7f9f7c0693e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:17.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 >> 172.21.15.37:0/754766360 conn(0x7f9f7c0fa380 msgr2=0x7f9f7c0fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:17.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 shutdown_connections 2024-04-15T22:51:17.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:17.773+0000 7f9f80ac4700 1 -- 172.21.15.37:0/754766360 wait complete. 2024-04-15T22:51:18.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:18 smithi037 ceph-mon[29191]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:18.559 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:18 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:18.579 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-15T22:51:18.579 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-15T22:51:18.788 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:51:19.271 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:19 smithi037 ceph-mon[29191]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:51:20.380 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:20 smithi037 ceph-mon[29191]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:21.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 -- 172.21.15.37:0/4012094905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 msgr2=0x7f6070102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 --2- 172.21.15.37:0/4012094905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070102c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6060009910 tx=0x7f6060009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 -- 172.21.15.37:0/4012094905 shutdown_connections 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 --2- 172.21.15.37:0/4012094905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070102c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 -- 172.21.15.37:0/4012094905 >> 172.21.15.37:0/4012094905 conn(0x7f60700fa2b0 msgr2=0x7f60700fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 -- 172.21.15.37:0/4012094905 shutdown_connections 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.315+0000 7f6074d45700 1 -- 172.21.15.37:0/4012094905 wait complete. 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f6074d45700 1 Processor -- start 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f6074d45700 1 -- start start 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f6074d45700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f6074d45700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6070195490 con 0x7f60701008b0 2024-04-15T22:51:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f606e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f606e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070194f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46504/0 (socket says 172.21.15.37:46504) 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.316+0000 7f606e59c700 1 -- 172.21.15.37:0/124377283 learned_addr learned my addr 172.21.15.37:0/124377283 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f606e59c700 1 -- 172.21.15.37:0/124377283 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60600095f0 con 0x7f60701008b0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f606e59c700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070194f50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f60600092e0 tx=0x7f6060009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f606001d070 con 0x7f60701008b0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6070195690 con 0x7f60701008b0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f606000be10 con 0x7f60701008b0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f606000f460 con 0x7f60701008b0 2024-04-15T22:51:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.317+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6070195b30 con 0x7f60701008b0 2024-04-15T22:51:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.318+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f606000f6d0 con 0x7f60701008b0 2024-04-15T22:51:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.318+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607018eb00 con 0x7f60701008b0 2024-04-15T22:51:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.318+0000 7f60677fe700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f605c038490 0x7f605c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.318+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f606004c690 con 0x7f60701008b0 2024-04-15T22:51:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.318+0000 7f6067fff700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f605c038490 0x7f605c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.319+0000 7f6067fff700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f605c038490 0x7f605c03a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6058006fd0 tx=0x7f6058006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.323+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6060005c00 con 0x7f60701008b0 2024-04-15T22:51:21.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.545+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f607002cc30 con 0x7f60701008b0 2024-04-15T22:51:21.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.960+0000 7f60677fe700 1 -- 172.21.15.37:0/124377283 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6060027030 con 0x7f60701008b0 2024-04-15T22:51:21.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.962+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f605c038490 msgr2=0x7f605c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:21.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.962+0000 7f6074d45700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f605c038490 0x7f605c03a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6058006fd0 tx=0x7f6058006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.962+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 msgr2=0x7f6070194f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.962+0000 7f6074d45700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070194f50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f60600092e0 tx=0x7f6060009e60 comp rx=0 tx=0).stop 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.963+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 shutdown_connections 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.963+0000 7f6074d45700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f605c038490 0x7f605c03a940 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.963+0000 7f6074d45700 1 --2- 172.21.15.37:0/124377283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60701008b0 0x7f6070194f50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.963+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 >> 172.21.15.37:0/124377283 conn(0x7f60700fa2b0 msgr2=0x7f60700faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.963+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 shutdown_connections 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:51:21.963+0000 7f6074d45700 1 -- 172.21.15.37:0/124377283 wait complete. 2024-04-15T22:51:21.960 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: Deploying daemon alertmanager.smithi037 on smithi037 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:22.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:21 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/124377283' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-15T22:51:22.762 INFO:tasks.cephadm:Adding mon.smithi037 on smithi037 2024-04-15T22:51:22.762 INFO:tasks.cephadm:Adding mon.smithi189 on smithi189 2024-04-15T22:51:22.762 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch apply mon '2;smithi037:172.21.15.37=smithi037;smithi189:172.21.15.189=smithi189' 2024-04-15T22:51:22.970 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:23.018 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:23.070 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:22 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/124377283' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T22:51:23.070 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:22 smithi037 ceph-mon[29191]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:51:24.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 -- 172.21.15.189:0/366301589 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 msgr2=0x7fce34102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:24.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 --2- 172.21.15.189:0/366301589 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34102c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fce30009910 tx=0x7fce30009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:24.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 -- 172.21.15.189:0/366301589 shutdown_connections 2024-04-15T22:51:24.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 --2- 172.21.15.189:0/366301589 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34102c90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:24.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 -- 172.21.15.189:0/366301589 >> 172.21.15.189:0/366301589 conn(0x7fce340fa2b0 msgr2=0x7fce340fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:24.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 -- 172.21.15.189:0/366301589 shutdown_connections 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.488+0000 7fce3ba1c700 1 -- 172.21.15.189:0/366301589 wait complete. 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce3ba1c700 1 Processor -- start 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce3ba1c700 1 -- start start 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce3ba1c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce3ba1c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce34195490 con 0x7fce341008b0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce397b8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce397b8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34194f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:52408/0 (socket says 172.21.15.189:52408) 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce397b8700 1 -- 172.21.15.189:0/4005570457 learned_addr learned my addr 172.21.15.189:0/4005570457 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce397b8700 1 -- 172.21.15.189:0/4005570457 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300095f0 con 0x7fce341008b0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.489+0000 7fce397b8700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34194f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fce30005b40 tx=0x7fce30004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.490+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce3001d070 con 0x7fce341008b0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.490+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fce3000be10 con 0x7fce341008b0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.490+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce34195690 con 0x7fce341008b0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.490+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce3000f460 con 0x7fce341008b0 2024-04-15T22:51:24.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.490+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce34195b30 con 0x7fce341008b0 2024-04-15T22:51:24.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.491+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fce3000f5c0 con 0x7fce341008b0 2024-04-15T22:51:24.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.491+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3418eb00 con 0x7fce341008b0 2024-04-15T22:51:24.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.491+0000 7fce2a7fc700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fce200384e0 0x7fce2003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:24.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.491+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fce3004c500 con 0x7fce341008b0 2024-04-15T22:51:24.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.491+0000 7fce38fb7700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fce200384e0 0x7fce2003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:24.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.491+0000 7fce38fb7700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fce200384e0 0x7fce2003a990 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fce24006fd0 tx=0x7fce24006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:24.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.496+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fce30020b10 con 0x7fce341008b0 2024-04-15T22:51:24.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:24 smithi037 ceph-mon[29191]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:24.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:24 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.695+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 --> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi037:172.21.15.37=smithi037;smithi189:172.21.15.189=smithi189", "target": ["mon-mgr", ""]}) v1 -- 0x7fce34061300 con 0x7fce200384e0 2024-04-15T22:51:24.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.701+0000 7fce2a7fc700 1 -- 172.21.15.189:0/4005570457 <== mgr.14162 v2:172.21.15.37:6800/712008483 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fce34061300 con 0x7fce200384e0 2024-04-15T22:51:24.697 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-04-15T22:51:24.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fce200384e0 msgr2=0x7fce2003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:24.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fce200384e0 0x7fce2003a990 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fce24006fd0 tx=0x7fce24006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:24.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 msgr2=0x7fce34194f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34194f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fce30005b40 tx=0x7fce30004740 comp rx=0 tx=0).stop 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 shutdown_connections 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fce200384e0 0x7fce2003a990 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 --2- 172.21.15.189:0/4005570457 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce341008b0 0x7fce34194f50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 >> 172.21.15.189:0/4005570457 conn(0x7fce340fa2b0 msgr2=0x7fce340faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 shutdown_connections 2024-04-15T22:51:24.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:24.703+0000 7fce3ba1c700 1 -- 172.21.15.189:0/4005570457 wait complete. 2024-04-15T22:51:25.265 DEBUG:teuthology.orchestra.run.smithi189:mon.smithi189> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@mon.smithi189.service 2024-04-15T22:51:25.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:25.267 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:25.554 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:25.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:25 smithi037 ceph-mon[29191]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi037:172.21.15.37=smithi037;smithi189:172.21.15.189=smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:51:25.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:25 smithi037 ceph-mon[29191]: Saving service mon spec with placement smithi037:172.21.15.37=smithi037;smithi189:172.21.15.189=smithi189;count:2 2024-04-15T22:51:25.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:25 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:25.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:25 smithi037 ceph-mon[29191]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 -- 172.21.15.189:0/1259616830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 msgr2=0x7f98840ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 --2- 172.21.15.189:0/1259616830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f98840ff130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9874009910 tx=0x7f9874009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 -- 172.21.15.189:0/1259616830 shutdown_connections 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 --2- 172.21.15.189:0/1259616830 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f98840ff130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 -- 172.21.15.189:0/1259616830 >> 172.21.15.189:0/1259616830 conn(0x7f98840fa2b0 msgr2=0x7f98840fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 -- 172.21.15.189:0/1259616830 shutdown_connections 2024-04-15T22:51:27.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.197+0000 7f988be9b700 1 -- 172.21.15.189:0/1259616830 wait complete. 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.198+0000 7f988be9b700 1 Processor -- start 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.198+0000 7f988be9b700 1 -- start start 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.198+0000 7f988be9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f9884194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.198+0000 7f988be9b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9884195500 con 0x7f98840fed20 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f9889c37700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f9884194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f9889c37700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f9884194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:38932/0 (socket says 172.21.15.189:38932) 2024-04-15T22:51:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f9889c37700 1 -- 172.21.15.189:0/2401709591 learned_addr learned my addr 172.21.15.189:0/2401709591 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f9889c37700 1 -- 172.21.15.189:0/2401709591 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98740095f0 con 0x7f98840fed20 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f9889c37700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f9884194fc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9874005b40 tx=0x7f9874004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f987401d070 con 0x7f98840fed20 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.199+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9884195700 con 0x7f98840fed20 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.200+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9884195ba0 con 0x7f98840fed20 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.200+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f987400be10 con 0x7f98840fed20 2024-04-15T22:51:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.200+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f987400f460 con 0x7f98840fed20 2024-04-15T22:51:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.200+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9868005320 con 0x7f98840fed20 2024-04-15T22:51:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.200+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f98740227c0 con 0x7f98840fed20 2024-04-15T22:51:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.201+0000 7f987affd700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9870038490 0x7f987003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.201+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f987404c2a0 con 0x7f98840fed20 2024-04-15T22:51:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.201+0000 7f9889436700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9870038490 0x7f987003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.201+0000 7f9889436700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9870038490 0x7f987003a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9880006fd0 tx=0x7f9880006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.206+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f987400b960 con 0x7f98840fed20 2024-04-15T22:51:27.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.468+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9868005190 con 0x7f98840fed20 2024-04-15T22:51:27.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.469+0000 7f987affd700 1 -- 172.21.15.189:0/2401709591 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9874027020 con 0x7f98840fed20 2024-04-15T22:51:27.465 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:27.465 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:27.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9870038490 msgr2=0x7f987003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:27.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9870038490 0x7f987003a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9880006fd0 tx=0x7f9880006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:27.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 msgr2=0x7f9884194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:27.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f9884194fc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9874005b40 tx=0x7f9874004740 comp rx=0 tx=0).stop 2024-04-15T22:51:27.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 shutdown_connections 2024-04-15T22:51:27.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9870038490 0x7f987003a940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:27.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 --2- 172.21.15.189:0/2401709591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98840fed20 0x7f9884194fc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:27.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 >> 172.21.15.189:0/2401709591 conn(0x7f98840fa2b0 msgr2=0x7f98840687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:27.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 shutdown_connections 2024-04-15T22:51:27.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:27.471+0000 7f988be9b700 1 -- 172.21.15.189:0/2401709591 wait complete. 2024-04-15T22:51:27.468 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:28.392 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:28 smithi037 ceph-mon[29191]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:28.392 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:28 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2401709591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:29.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:29.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:29.228 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:29.275 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.478 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.479 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:51:30.479 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:30 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:30.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 -- 172.21.15.189:0/692011210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 msgr2=0x7f739c0fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:30.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 --2- 172.21.15.189:0/692011210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c0fe4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f739400b0a0 tx=0x7f739400b3b0 comp rx=0 tx=0).stop 2024-04-15T22:51:30.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 -- 172.21.15.189:0/692011210 shutdown_connections 2024-04-15T22:51:30.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 --2- 172.21.15.189:0/692011210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c0fe4f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:30.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 -- 172.21.15.189:0/692011210 >> 172.21.15.189:0/692011210 conn(0x7f739c0f9a50 msgr2=0x7f739c0fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 -- 172.21.15.189:0/692011210 shutdown_connections 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.685+0000 7f73a410d700 1 -- 172.21.15.189:0/692011210 wait complete. 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a410d700 1 Processor -- start 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a410d700 1 -- start start 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a410d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a410d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1954f0 con 0x7f739c0fe0e0 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a1ea9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a1ea9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c194fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:38946/0 (socket says 172.21.15.189:38946) 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.686+0000 7f73a1ea9700 1 -- 172.21.15.189:0/1682369097 learned_addr learned my addr 172.21.15.189:0/1682369097 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f73a1ea9700 1 -- 172.21.15.189:0/1682369097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394009d00 con 0x7f739c0fe0e0 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f73a1ea9700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c194fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f739400b830 tx=0x7f739400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f739400e040 con 0x7f739c0fe0e0 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c1956f0 con 0x7f739c0fe0e0 2024-04-15T22:51:30.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c195b10 con 0x7f739c0fe0e0 2024-04-15T22:51:30.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7394012af0 con 0x7f739c0fe0e0 2024-04-15T22:51:30.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.687+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f739401d940 con 0x7f739c0fe0e0 2024-04-15T22:51:30.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.688+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f739401dbb0 con 0x7f739c0fe0e0 2024-04-15T22:51:30.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.688+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7388005320 con 0x7f739c0fe0e0 2024-04-15T22:51:30.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.688+0000 7f739affd700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7384038490 0x7f738403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:30.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.688+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f739404c990 con 0x7f739c0fe0e0 2024-04-15T22:51:30.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.688+0000 7f73a16a8700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7384038490 0x7f738403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:30.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.689+0000 7f73a16a8700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7384038490 0x7f738403a940 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7390006fd0 tx=0x7f7390006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:30.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.693+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f73940263c0 con 0x7f739c0fe0e0 2024-04-15T22:51:30.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.959+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7388005190 con 0x7f739c0fe0e0 2024-04-15T22:51:30.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.959+0000 7f739affd700 1 -- 172.21.15.189:0/1682369097 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f739401de60 con 0x7f739c0fe0e0 2024-04-15T22:51:30.956 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:30.956 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7384038490 msgr2=0x7f738403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7384038490 0x7f738403a940 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7390006fd0 tx=0x7f7390006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 msgr2=0x7f739c194fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c194fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f739400b830 tx=0x7f739400bd40 comp rx=0 tx=0).stop 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 shutdown_connections 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f7384038490 0x7f738403a940 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 --2- 172.21.15.189:0/1682369097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c0fe0e0 0x7f739c194fb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 >> 172.21.15.189:0/1682369097 conn(0x7f739c0f9a50 msgr2=0x7f739c0fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 shutdown_connections 2024-04-15T22:51:30.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:30.962+0000 7f73a410d700 1 -- 172.21.15.189:0/1682369097 wait complete. 2024-04-15T22:51:30.961 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:31.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:31 smithi037 ceph-mon[29191]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-15T22:51:31.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:31 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:51:31.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:31 smithi037 ceph-mon[29191]: Deploying daemon grafana.smithi037 on smithi037 2024-04-15T22:51:31.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:31 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/1682369097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:32.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:32 smithi037 ceph-mon[29191]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:32.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:32.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:32.911 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:32.959 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.432+0000 7f6476924700 1 -- 172.21.15.189:0/3481271260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 msgr2=0x7f64700ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.432+0000 7f6476924700 1 --2- 172.21.15.189:0/3481271260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f64700ff130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6460009910 tx=0x7f6460009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 -- 172.21.15.189:0/3481271260 shutdown_connections 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 --2- 172.21.15.189:0/3481271260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f64700ff130 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 -- 172.21.15.189:0/3481271260 >> 172.21.15.189:0/3481271260 conn(0x7f64700fa2b0 msgr2=0x7f64700fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 -- 172.21.15.189:0/3481271260 shutdown_connections 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 -- 172.21.15.189:0/3481271260 wait complete. 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 Processor -- start 2024-04-15T22:51:34.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.433+0000 7f6476924700 1 -- start start 2024-04-15T22:51:34.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f6476924700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f6470194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:34.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f6476924700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64701954f0 con 0x7f64700fed20 2024-04-15T22:51:34.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f6470194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:34.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f6470194fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:38964/0 (socket says 172.21.15.189:38964) 2024-04-15T22:51:34.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646ffff700 1 -- 172.21.15.189:0/1704658180 learned_addr learned my addr 172.21.15.189:0/1704658180 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:34.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646ffff700 1 -- 172.21.15.189:0/1704658180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64600095f0 con 0x7f64700fed20 2024-04-15T22:51:34.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646ffff700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f6470194fb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6460005b40 tx=0x7f6460004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:34.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f646001d070 con 0x7f64700fed20 2024-04-15T22:51:34.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64701956f0 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f646000be10 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f646000f460 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.434+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6470195b90 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.435+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f646000b960 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.435+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647018ea10 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.435+0000 7f646d7fa700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6458038490 0x7f645803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.435+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f646004c4a0 con 0x7f64700fed20 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.436+0000 7f646f7fe700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6458038490 0x7f645803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:34.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.436+0000 7f646f7fe700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6458038490 0x7f645803a940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6464006fd0 tx=0x7f6464006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:34.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.440+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f646002a360 con 0x7f64700fed20 2024-04-15T22:51:34.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:34 smithi037 ceph-mon[29191]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:34.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:34 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:34.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.704+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f647002cc30 con 0x7f64700fed20 2024-04-15T22:51:34.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.705+0000 7f646d7fa700 1 -- 172.21.15.189:0/1704658180 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6460027030 con 0x7f64700fed20 2024-04-15T22:51:34.702 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:34.703 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.707+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6458038490 msgr2=0x7f645803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.707+0000 7f6476924700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6458038490 0x7f645803a940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6464006fd0 tx=0x7f6464006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.707+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 msgr2=0x7f6470194fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.707+0000 7f6476924700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f6470194fb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6460005b40 tx=0x7f6460004740 comp rx=0 tx=0).stop 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.708+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 shutdown_connections 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.708+0000 7f6476924700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6458038490 0x7f645803a940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.708+0000 7f6476924700 1 --2- 172.21.15.189:0/1704658180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64700fed20 0x7f6470194fb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:34.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.708+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 >> 172.21.15.189:0/1704658180 conn(0x7f64700fa2b0 msgr2=0x7f64700690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:34.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.708+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 shutdown_connections 2024-04-15T22:51:34.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:34.708+0000 7f6476924700 1 -- 172.21.15.189:0/1704658180 wait complete. 2024-04-15T22:51:34.706 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:35.484 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:35 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/1704658180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:36.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:36.250 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:36.455 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:36.501 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:36.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:36 smithi037 ceph-mon[29191]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:37.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 -- 172.21.15.189:0/1513777378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 msgr2=0x7fae340ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 --2- 172.21.15.189:0/1513777378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae340ff130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae24009910 tx=0x7fae24009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 -- 172.21.15.189:0/1513777378 shutdown_connections 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 --2- 172.21.15.189:0/1513777378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae340ff130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 -- 172.21.15.189:0/1513777378 >> 172.21.15.189:0/1513777378 conn(0x7fae340fa2b0 msgr2=0x7fae340fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 -- 172.21.15.189:0/1513777378 shutdown_connections 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.908+0000 7fae38883700 1 -- 172.21.15.189:0/1513777378 wait complete. 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae38883700 1 Processor -- start 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae38883700 1 -- start start 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae38883700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae34194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae38883700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae34195500 con 0x7fae340fed20 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae3259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae34194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae3259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae34194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:42826/0 (socket says 172.21.15.189:42826) 2024-04-15T22:51:37.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.909+0000 7fae3259c700 1 -- 172.21.15.189:0/4131827176 learned_addr learned my addr 172.21.15.189:0/4131827176 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae3259c700 1 -- 172.21.15.189:0/4131827176 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae240095f0 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae3259c700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae34194fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae24005950 tx=0x7fae24009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae2401d070 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fae2400be10 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae34195700 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae2400f460 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.910+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae34195ba0 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.911+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fae2400f5c0 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.911+0000 7fae2b7fe700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fae200384e0 0x7fae2003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.911+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fae2404c520 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.911+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3418ea10 con 0x7fae340fed20 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.911+0000 7fae2bfff700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fae200384e0 0x7fae2003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:37.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.912+0000 7fae2bfff700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fae200384e0 0x7fae2003a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae1c006fd0 tx=0x7fae1c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:37.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:37.917+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fae24027070 con 0x7fae340fed20 2024-04-15T22:51:38.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.183+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae3402cc30 con 0x7fae340fed20 2024-04-15T22:51:38.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.183+0000 7fae2b7fe700 1 -- 172.21.15.189:0/4131827176 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fae2402c440 con 0x7fae340fed20 2024-04-15T22:51:38.180 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:38.181 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:38.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fae200384e0 msgr2=0x7fae2003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fae200384e0 0x7fae2003a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae1c006fd0 tx=0x7fae1c006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 msgr2=0x7fae34194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae34194fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae24005950 tx=0x7fae24009e60 comp rx=0 tx=0).stop 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 shutdown_connections 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fae200384e0 0x7fae2003a990 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 --2- 172.21.15.189:0/4131827176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae340fed20 0x7fae34194fc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 >> 172.21.15.189:0/4131827176 conn(0x7fae340fa2b0 msgr2=0x7fae340687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 shutdown_connections 2024-04-15T22:51:38.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:38.185+0000 7fae38883700 1 -- 172.21.15.189:0/4131827176 wait complete. 2024-04-15T22:51:38.184 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:38.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:38 smithi037 ceph-mon[29191]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:39.408 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:39 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/4131827176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:39.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:39.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:40.010 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:40.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:40 smithi037 ceph-mon[29191]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:41.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/2648778715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 msgr2=0x7fdfdc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:41.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 --2- 172.21.15.189:0/2648778715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc0731e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009910 tx=0x7fdfd8009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/2648778715 shutdown_connections 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 --2- 172.21.15.189:0/2648778715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc0731e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/2648778715 >> 172.21.15.189:0/2648778715 conn(0x7fdfdc0fb3d0 msgr2=0x7fdfdc0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/2648778715 shutdown_connections 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/2648778715 wait complete. 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 Processor -- start 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- start start 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc19b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.554+0000 7fdfe3ce7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc19ba90 con 0x7fdfdc074d80 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe1a83700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc19b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe1a83700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc19b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:42840/0 (socket says 172.21.15.189:42840) 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe1a83700 1 -- 172.21.15.189:0/3535329111 learned_addr learned my addr 172.21.15.189:0/3535329111 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe1a83700 1 -- 172.21.15.189:0/3535329111 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd80095f0 con 0x7fdfdc074d80 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe1a83700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc19b550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8005b40 tx=0x7fdfd8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd801d070 con 0x7fdfdc074d80 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc19bc90 con 0x7fdfdc074d80 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdfd800be10 con 0x7fdfdc074d80 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd800f460 con 0x7fdfdc074d80 2024-04-15T22:51:41.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.555+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc19c130 con 0x7fdfdc074d80 2024-04-15T22:51:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.556+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc195100 con 0x7fdfdc074d80 2024-04-15T22:51:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.556+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fdfd80227e0 con 0x7fdfdc074d80 2024-04-15T22:51:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.557+0000 7fdfd2ffd700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fdfc8038490 0x7fdfc803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.557+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdfd804c4f0 con 0x7fdfdc074d80 2024-04-15T22:51:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.557+0000 7fdfe1282700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fdfc8038490 0x7fdfc803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.557+0000 7fdfe1282700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fdfc8038490 0x7fdfc803a940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006fd0 tx=0x7fdfcc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:41.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.562+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdfd802bba0 con 0x7fdfdc074d80 2024-04-15T22:51:41.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.826+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfdc02d050 con 0x7fdfdc074d80 2024-04-15T22:51:41.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.827+0000 7fdfd2ffd700 1 -- 172.21.15.189:0/3535329111 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fdfd8027070 con 0x7fdfdc074d80 2024-04-15T22:51:41.824 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:41.824 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fdfc8038490 msgr2=0x7fdfc803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fdfc8038490 0x7fdfc803a940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006fd0 tx=0x7fdfcc006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 msgr2=0x7fdfdc19b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc19b550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8005b40 tx=0x7fdfd8004740 comp rx=0 tx=0).stop 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 shutdown_connections 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fdfc8038490 0x7fdfc803a940 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 --2- 172.21.15.189:0/3535329111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdfdc074d80 0x7fdfdc19b550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 >> 172.21.15.189:0/3535329111 conn(0x7fdfdc0fb3d0 msgr2=0x7fdfdc0fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 shutdown_connections 2024-04-15T22:51:41.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:41.829+0000 7fdfe3ce7700 1 -- 172.21.15.189:0/3535329111 wait complete. 2024-04-15T22:51:41.827 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:42.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:42 smithi037 ceph-mon[29191]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:42.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:42 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/3535329111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:43.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:43.418 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:43.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:43 smithi037 ceph-mon[29191]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:43.624 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:43.670 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:45.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.089+0000 7f820b10e700 1 -- 172.21.15.189:0/3080637095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 msgr2=0x7f82040ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:45.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.089+0000 7f820b10e700 1 --2- 172.21.15.189:0/3080637095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f82040ff130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b0a0 tx=0x7f81f400b3b0 comp rx=0 tx=0).stop 2024-04-15T22:51:45.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.089+0000 7f820b10e700 1 -- 172.21.15.189:0/3080637095 shutdown_connections 2024-04-15T22:51:45.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.089+0000 7f820b10e700 1 --2- 172.21.15.189:0/3080637095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f82040ff130 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.089+0000 7f820b10e700 1 -- 172.21.15.189:0/3080637095 >> 172.21.15.189:0/3080637095 conn(0x7f82040fa2b0 msgr2=0x7f82040fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.090+0000 7f820b10e700 1 -- 172.21.15.189:0/3080637095 shutdown_connections 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.090+0000 7f820b10e700 1 -- 172.21.15.189:0/3080637095 wait complete. 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.090+0000 7f820b10e700 1 Processor -- start 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.090+0000 7f820b10e700 1 -- start start 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.090+0000 7f820b10e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f8204069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.090+0000 7f820b10e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8204197cf0 con 0x7f82040fed20 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8208eaa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f8204069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8208eaa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f8204069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:59090/0 (socket says 172.21.15.189:59090) 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8208eaa700 1 -- 172.21.15.189:0/905782062 learned_addr learned my addr 172.21.15.189:0/905782062 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8208eaa700 1 -- 172.21.15.189:0/905782062 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4009d00 con 0x7f82040fed20 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8208eaa700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f8204069380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b830 tx=0x7f81f400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:45.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81f400e040 con 0x7f82040fed20 2024-04-15T22:51:45.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f81f4012af0 con 0x7f82040fed20 2024-04-15T22:51:45.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8204197e30 con 0x7f82040fed20 2024-04-15T22:51:45.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81f401d9f0 con 0x7f82040fed20 2024-04-15T22:51:45.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.091+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82041981f0 con 0x7f82040fed20 2024-04-15T22:51:45.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.092+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820418da00 con 0x7f82040fed20 2024-04-15T22:51:45.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.092+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f81f4012c60 con 0x7f82040fed20 2024-04-15T22:51:45.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.093+0000 7f8201ffb700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f81f0038490 0x7f81f003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:45.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.093+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f81f404c8d0 con 0x7f82040fed20 2024-04-15T22:51:45.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.093+0000 7f8203fff700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f81f0038490 0x7f81f003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:45.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.093+0000 7f8203fff700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f81f0038490 0x7f81f003a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006fd0 tx=0x7f81ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:45.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.098+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f81f401c5c0 con 0x7f82040fed20 2024-04-15T22:51:45.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.363+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f820402cc30 con 0x7f82040fed20 2024-04-15T22:51:45.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.364+0000 7f8201ffb700 1 -- 172.21.15.189:0/905782062 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f81f401c3e0 con 0x7f82040fed20 2024-04-15T22:51:45.361 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:45.361 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:45.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f81f0038490 msgr2=0x7f81f003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:45.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f81f0038490 0x7f81f003a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006fd0 tx=0x7f81ec006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:45.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 msgr2=0x7f8204069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:45.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f8204069380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b830 tx=0x7f81f400bd40 comp rx=0 tx=0).stop 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 shutdown_connections 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f81f0038490 0x7f81f003a940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 --2- 172.21.15.189:0/905782062 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82040fed20 0x7f8204069380 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 >> 172.21.15.189:0/905782062 conn(0x7f82040fa2b0 msgr2=0x7f82040faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 shutdown_connections 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:45.366+0000 7f820b10e700 1 -- 172.21.15.189:0/905782062 wait complete. 2024-04-15T22:51:45.364 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:46.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:46 smithi037 ceph-mon[29191]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:46.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:46 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/905782062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:46.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:46.920 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:47.126 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:48.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:48 smithi037 ceph-mon[29191]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:48.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 -- 172.21.15.189:0/1333172973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 msgr2=0x7f576c100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 --2- 172.21.15.189:0/1333172973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c100280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f575c009910 tx=0x7f575c009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 -- 172.21.15.189:0/1333172973 shutdown_connections 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 --2- 172.21.15.189:0/1333172973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c100280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 -- 172.21.15.189:0/1333172973 >> 172.21.15.189:0/1333172973 conn(0x7f576c0fb470 msgr2=0x7f576c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 -- 172.21.15.189:0/1333172973 shutdown_connections 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.649+0000 7f5772295700 1 -- 172.21.15.189:0/1333172973 wait complete. 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f5772295700 1 Processor -- start 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f5772295700 1 -- start start 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f5772295700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f5772295700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1954e0 con 0x7f576c0ffe70 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f576bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:48.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f576bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c194fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:59104/0 (socket says 172.21.15.189:59104) 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f576bfff700 1 -- 172.21.15.189:0/3481446898 learned_addr learned my addr 172.21.15.189:0/3481446898 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.650+0000 7f576bfff700 1 -- 172.21.15.189:0/3481446898 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0095f0 con 0x7f576c0ffe70 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.651+0000 7f576bfff700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c194fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f575c005b40 tx=0x7f575c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.651+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575c01d070 con 0x7f576c0ffe70 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.651+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f575c00be10 con 0x7f576c0ffe70 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.651+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c1956e0 con 0x7f576c0ffe70 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.651+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575c00f460 con 0x7f576c0ffe70 2024-04-15T22:51:48.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.651+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c195b80 con 0x7f576c0ffe70 2024-04-15T22:51:48.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.652+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c18ea70 con 0x7f576c0ffe70 2024-04-15T22:51:48.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.652+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f575c0227e0 con 0x7f576c0ffe70 2024-04-15T22:51:48.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.652+0000 7f5769ffb700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f5758038490 0x7f575803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:48.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.652+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f575c04c3f0 con 0x7f576c0ffe70 2024-04-15T22:51:48.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.652+0000 7f5763fff700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f5758038490 0x7f575803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:48.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.653+0000 7f5763fff700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f5758038490 0x7f575803a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5754006fd0 tx=0x7f5754006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:48.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.657+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f575c02bba0 con 0x7f576c0ffe70 2024-04-15T22:51:48.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.922+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f576c02cc30 con 0x7f576c0ffe70 2024-04-15T22:51:48.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.923+0000 7f5769ffb700 1 -- 172.21.15.189:0/3481446898 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f575c027030 con 0x7f576c0ffe70 2024-04-15T22:51:48.920 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:48.920 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f5758038490 msgr2=0x7f575803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f5758038490 0x7f575803a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5754006fd0 tx=0x7f5754006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 msgr2=0x7f576c194fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c194fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f575c005b40 tx=0x7f575c004740 comp rx=0 tx=0).stop 2024-04-15T22:51:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 shutdown_connections 2024-04-15T22:51:48.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f5758038490 0x7f575803a940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:48.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 --2- 172.21.15.189:0/3481446898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f576c0ffe70 0x7f576c194fa0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:48.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 >> 172.21.15.189:0/3481446898 conn(0x7f576c0fb470 msgr2=0x7f576c0fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:48.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 shutdown_connections 2024-04-15T22:51:48.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:48.925+0000 7f5772295700 1 -- 172.21.15.189:0/3481446898 wait complete. 2024-04-15T22:51:48.923 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:49.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:49 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/3481446898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:50.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:50 smithi037 ceph-mon[29191]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:50.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:50.448 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:50.652 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:50.699 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:52.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2366777859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 msgr2=0x7f91a81007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 --2- 172.21.15.189:0/2366777859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a81007b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9198009910 tx=0x7f9198009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2366777859 shutdown_connections 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 --2- 172.21.15.189:0/2366777859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a81007b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2366777859 >> 172.21.15.189:0/2366777859 conn(0x7f91a80f9a50 msgr2=0x7f91a80fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2366777859 shutdown_connections 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.123+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2366777859 wait complete. 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91ae3ad700 1 Processor -- start 2024-04-15T22:51:52.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91ae3ad700 1 -- start start 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91ae3ad700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a8194f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91ae3ad700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81954c0 con 0x7f91a81003a0 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91a7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a8194f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91a7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a8194f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:59122/0 (socket says 172.21.15.189:59122) 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.124+0000 7f91a7fff700 1 -- 172.21.15.189:0/2962853318 learned_addr learned my addr 172.21.15.189:0/2962853318 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91a7fff700 1 -- 172.21.15.189:0/2962853318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980095f0 con 0x7f91a81003a0 2024-04-15T22:51:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91a7fff700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a8194f80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9198005b40 tx=0x7f9198004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919801d070 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f919800be10 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a81956c0 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919800f460 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.125+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a8195b60 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.126+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f919800b960 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.126+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a818eac0 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.126+0000 7f91a57fa700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9190038490 0x7f919003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.126+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f919804c4c0 con 0x7f91a81003a0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.126+0000 7f91a77fe700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9190038490 0x7f919003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.127+0000 7f91a77fe700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9190038490 0x7f919003a940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f919c006fd0 tx=0x7f919c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:52.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.135+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f919802a360 con 0x7f91a81003a0 2024-04-15T22:51:52.169 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:52.169 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:52.169 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:52.169 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:52.169 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:52.170 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:52.170 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:52 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:52.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.399+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91a802cc30 con 0x7f91a81003a0 2024-04-15T22:51:52.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.400+0000 7f91a57fa700 1 -- 172.21.15.189:0/2962853318 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9198027030 con 0x7f91a81003a0 2024-04-15T22:51:52.397 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:52.397 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:52.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9190038490 msgr2=0x7f919003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:52.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9190038490 0x7f919003a940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f919c006fd0 tx=0x7f919c006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:52.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 msgr2=0x7f91a8194f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:52.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a8194f80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9198005b40 tx=0x7f9198004740 comp rx=0 tx=0).stop 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 shutdown_connections 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f9190038490 0x7f919003a940 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 --2- 172.21.15.189:0/2962853318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a81003a0 0x7f91a8194f80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 >> 172.21.15.189:0/2962853318 conn(0x7f91a80f9a50 msgr2=0x7f91a80fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 shutdown_connections 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:52.402+0000 7f91ae3ad700 1 -- 172.21.15.189:0/2962853318 wait complete. 2024-04-15T22:51:52.400 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:53.506 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:53 smithi037 ceph-mon[29191]: Deploying daemon prometheus.smithi037 on smithi037 2024-04-15T22:51:53.507 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:53 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2962853318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:53.507 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:53 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:51:54.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:54.028 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:54.232 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:54.279 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:54.530 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:54 smithi037 ceph-mon[29191]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 -- 172.21.15.189:0/267297208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 msgr2=0x7f86480fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 --2- 172.21.15.189:0/267297208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f86480fe4f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f863800b0a0 tx=0x7f863800b3b0 comp rx=0 tx=0).stop 2024-04-15T22:51:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 -- 172.21.15.189:0/267297208 shutdown_connections 2024-04-15T22:51:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 --2- 172.21.15.189:0/267297208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f86480fe4f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 -- 172.21.15.189:0/267297208 >> 172.21.15.189:0/267297208 conn(0x7f86480f9a50 msgr2=0x7f86480fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 -- 172.21.15.189:0/267297208 shutdown_connections 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.761+0000 7f864f40e700 1 -- 172.21.15.189:0/267297208 wait complete. 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864f40e700 1 Processor -- start 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864f40e700 1 -- start start 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864f40e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f8648100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864f40e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8648101000 con 0x7f86480fe0e0 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864d1aa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f8648100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864d1aa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f8648100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:45388/0 (socket says 172.21.15.189:45388) 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864d1aa700 1 -- 172.21.15.189:0/846534418 learned_addr learned my addr 172.21.15.189:0/846534418 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.762+0000 7f864d1aa700 1 -- 172.21.15.189:0/846534418 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638009d00 con 0x7f86480fe0e0 2024-04-15T22:51:55.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.763+0000 7f864d1aa700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f8648100ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f863800b830 tx=0x7f863800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:55.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.763+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f863800e040 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.763+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86480ff110 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.763+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8638012af0 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.763+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f863801d9f0 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.763+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86480ff530 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.764+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f863801dcb0 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.764+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8648062380 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.764+0000 7f86467fc700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f8630038490 0x7f863003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.764+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f863804cb40 con 0x7f86480fe0e0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.764+0000 7f864c9a9700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f8630038490 0x7f863003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:55.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.765+0000 7f864c9a9700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f8630038490 0x7f863003a940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:55.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:55.770+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f863801aba0 con 0x7f86480fe0e0 2024-04-15T22:51:56.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.038+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f864818e4e0 con 0x7f86480fe0e0 2024-04-15T22:51:56.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.039+0000 7f86467fc700 1 -- 172.21.15.189:0/846534418 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f863801aba0 con 0x7f86480fe0e0 2024-04-15T22:51:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:56.035 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f8630038490 msgr2=0x7f863003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f8630038490 0x7f863003a940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 msgr2=0x7f8648100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f8648100ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f863800b830 tx=0x7f863800bd40 comp rx=0 tx=0).stop 2024-04-15T22:51:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 shutdown_connections 2024-04-15T22:51:56.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f8630038490 0x7f863003a940 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:56.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 --2- 172.21.15.189:0/846534418 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86480fe0e0 0x7f8648100ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:56.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 >> 172.21.15.189:0/846534418 conn(0x7f86480f9a50 msgr2=0x7f86480fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:56.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.041+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 shutdown_connections 2024-04-15T22:51:56.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:56.042+0000 7f864f40e700 1 -- 172.21.15.189:0/846534418 wait complete. 2024-04-15T22:51:56.038 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:51:56.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:56 smithi037 ceph-mon[29191]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:56.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:56 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/846534418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:51:57.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:51:57.624 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:51:57.831 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:57.877 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:51:58.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:51:58 smithi037 ceph-mon[29191]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 -- 172.21.15.189:0/162706738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 msgr2=0x7f3cb8100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 --2- 172.21.15.189:0/162706738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8100230 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009910 tx=0x7f3ca8009c20 comp rx=0 tx=0).stop 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 -- 172.21.15.189:0/162706738 shutdown_connections 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 --2- 172.21.15.189:0/162706738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8100230 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 -- 172.21.15.189:0/162706738 >> 172.21.15.189:0/162706738 conn(0x7f3cb80fb3d0 msgr2=0x7f3cb80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 -- 172.21.15.189:0/162706738 shutdown_connections 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.273+0000 7f3cbf566700 1 -- 172.21.15.189:0/162706738 wait complete. 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.274+0000 7f3cbf566700 1 Processor -- start 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.274+0000 7f3cbf566700 1 -- start start 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.274+0000 7f3cbf566700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.274+0000 7f3cbf566700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81954a0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.274+0000 7f3cbd302700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:59.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cbd302700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:45400/0 (socket says 172.21.15.189:45400) 2024-04-15T22:51:59.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cbd302700 1 -- 172.21.15.189:0/3259236970 learned_addr learned my addr 172.21.15.189:0/3259236970 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cbd302700 1 -- 172.21.15.189:0/3259236970 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca80095f0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cbd302700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8194f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80092e0 tx=0x7f3ca8009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca801d070 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ca800be10 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb81956a0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca800f460 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.275+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb8195b40 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.276+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f3ca800f6d0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.276+0000 7f3cae7fc700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3ca4038490 0x7f3ca403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.276+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb81008b0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.276+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ca804c380 con 0x7f3cb80ffe20 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.276+0000 7f3cbcb01700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3ca4038490 0x7f3ca403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:51:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.277+0000 7f3cbcb01700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3ca4038490 0x7f3ca403a940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4006fd0 tx=0x7f3cb4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:51:59.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.282+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3cb81008b0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.547+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cb81008b0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.548+0000 7f3cae7fc700 1 -- 172.21.15.189:0/3259236970 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3cb81008b0 con 0x7f3cb80ffe20 2024-04-15T22:51:59.544 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:51:59.544 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3ca4038490 msgr2=0x7f3ca403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3ca4038490 0x7f3ca403a940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4006fd0 tx=0x7f3cb4006e40 comp rx=0 tx=0).stop 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 msgr2=0x7f3cb8194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8194f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80092e0 tx=0x7f3ca8009e60 comp rx=0 tx=0).stop 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 shutdown_connections 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f3ca4038490 0x7f3ca403a940 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 --2- 172.21.15.189:0/3259236970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cb80ffe20 0x7f3cb8194f60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:51:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 >> 172.21.15.189:0/3259236970 conn(0x7f3cb80fb3d0 msgr2=0x7f3cb80fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:51:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 shutdown_connections 2024-04-15T22:51:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:51:59.550+0000 7f3cbf566700 1 -- 172.21.15.189:0/3259236970 wait complete. 2024-04-15T22:51:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:00.517 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:00 smithi037 ceph-mon[29191]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:00.518 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:00 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/3259236970' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:01.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:01.119 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:01.370 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:02.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:02 smithi037 ceph-mon[29191]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.883+0000 7f444772e700 1 -- 172.21.15.189:0/3273126075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 msgr2=0x7f44400ff130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.883+0000 7f444772e700 1 --2- 172.21.15.189:0/3273126075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400ff130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4430009910 tx=0x7f4430009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.883+0000 7f444772e700 1 -- 172.21.15.189:0/3273126075 shutdown_connections 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.883+0000 7f444772e700 1 --2- 172.21.15.189:0/3273126075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400ff130 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.883+0000 7f444772e700 1 -- 172.21.15.189:0/3273126075 >> 172.21.15.189:0/3273126075 conn(0x7f44400fa2b0 msgr2=0x7f44400fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.884+0000 7f444772e700 1 -- 172.21.15.189:0/3273126075 shutdown_connections 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.884+0000 7f444772e700 1 -- 172.21.15.189:0/3273126075 wait complete. 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.884+0000 7f444772e700 1 Processor -- start 2024-04-15T22:52:02.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.884+0000 7f444772e700 1 -- start start 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.884+0000 7f444772e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.884+0000 7f444772e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44400fc7a0 con 0x7f44400fed20 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44454ca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44454ca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:45418/0 (socket says 172.21.15.189:45418) 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44454ca700 1 -- 172.21.15.189:0/897810574 learned_addr learned my addr 172.21.15.189:0/897810574 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44454ca700 1 -- 172.21.15.189:0/897810574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300095f0 con 0x7f44400fed20 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44454ca700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400fc260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4430005b40 tx=0x7f4430004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f443001d070 con 0x7f44400fed20 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44400fa8b0 con 0x7f44400fed20 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f443000be10 con 0x7f44400fed20 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f443000f460 con 0x7f44400fed20 2024-04-15T22:52:02.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.885+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44400fad50 con 0x7f44400fed20 2024-04-15T22:52:02.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.886+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440062380 con 0x7f44400fed20 2024-04-15T22:52:02.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.886+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f44300227e0 con 0x7f44400fed20 2024-04-15T22:52:02.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.887+0000 7f44367fc700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f442c038490 0x7f442c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:02.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.887+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f443004c3f0 con 0x7f44400fed20 2024-04-15T22:52:02.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.887+0000 7f4444cc9700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f442c038490 0x7f442c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:02.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.887+0000 7f4444cc9700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f442c038490 0x7f442c03a940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f443c006fd0 tx=0x7f443c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:02.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:02.892+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f443002bba0 con 0x7f44400fed20 2024-04-15T22:52:03.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.161+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f444018e3f0 con 0x7f44400fed20 2024-04-15T22:52:03.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.162+0000 7f44367fc700 1 -- 172.21.15.189:0/897810574 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4430027030 con 0x7f44400fed20 2024-04-15T22:52:03.157 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:03.158 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:03.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f442c038490 msgr2=0x7f442c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:03.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f442c038490 0x7f442c03a940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f443c006fd0 tx=0x7f443c006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:03.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 msgr2=0x7f44400fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400fc260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4430005b40 tx=0x7f4430004740 comp rx=0 tx=0).stop 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 shutdown_connections 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f442c038490 0x7f442c03a940 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 --2- 172.21.15.189:0/897810574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44400fed20 0x7f44400fc260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 >> 172.21.15.189:0/897810574 conn(0x7f44400fa2b0 msgr2=0x7f44400687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 shutdown_connections 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:03.164+0000 7f444772e700 1 -- 172.21.15.189:0/897810574 wait complete. 2024-04-15T22:52:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:03.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:03 smithi037 ceph-mon[29191]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:03.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:03 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/897810574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:04.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:04.787 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:04.993 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:05.042 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- 172.21.15.189:0/1704427475 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 msgr2=0x7f8254100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 --2- 172.21.15.189:0/1704427475 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f8254100720 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8244009910 tx=0x7f8244009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- 172.21.15.189:0/1704427475 shutdown_connections 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 --2- 172.21.15.189:0/1704427475 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f8254100720 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- 172.21.15.189:0/1704427475 >> 172.21.15.189:0/1704427475 conn(0x7f82540f9af0 msgr2=0x7f82540fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- 172.21.15.189:0/1704427475 shutdown_connections 2024-04-15T22:52:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- 172.21.15.189:0/1704427475 wait complete. 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 Processor -- start 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- start start 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f82540fba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.497+0000 7f825c00a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82540fbfc0 con 0x7f8254100310 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f8259da6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f82540fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f8259da6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f82540fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:55622/0 (socket says 172.21.15.189:55622) 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f8259da6700 1 -- 172.21.15.189:0/2348881973 learned_addr learned my addr 172.21.15.189:0/2348881973 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f8259da6700 1 -- 172.21.15.189:0/2348881973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440095f0 con 0x7f8254100310 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f8259da6700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f82540fba80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8244005b40 tx=0x7f8244004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f824401d070 con 0x7f8254100310 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f824400be10 con 0x7f8254100310 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82540fa0d0 con 0x7f8254100310 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f824400f460 con 0x7f8254100310 2024-04-15T22:52:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.498+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82540fa570 con 0x7f8254100310 2024-04-15T22:52:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.499+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f824400b960 con 0x7f8254100310 2024-04-15T22:52:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.499+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8254062380 con 0x7f8254100310 2024-04-15T22:52:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.499+0000 7f824affd700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f82400384e0 0x7f824003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.499+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f824404c470 con 0x7f8254100310 2024-04-15T22:52:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.500+0000 7f82595a5700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f82400384e0 0x7f824003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.500+0000 7f82595a5700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f82400384e0 0x7f824003a990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8250006fd0 tx=0x7f8250006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:06.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.505+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f824402bb60 con 0x7f8254100310 2024-04-15T22:52:06.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:06 smithi037 ceph-mon[29191]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:06.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.768+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f825418e370 con 0x7f8254100310 2024-04-15T22:52:06.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.768+0000 7f824affd700 1 -- 172.21.15.189:0/2348881973 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8244027070 con 0x7f8254100310 2024-04-15T22:52:06.764 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:06.764 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:06.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f82400384e0 msgr2=0x7f824003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f82400384e0 0x7f824003a990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8250006fd0 tx=0x7f8250006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 msgr2=0x7f82540fba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f82540fba80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8244005b40 tx=0x7f8244004740 comp rx=0 tx=0).stop 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 shutdown_connections 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f82400384e0 0x7f824003a990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 --2- 172.21.15.189:0/2348881973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8254100310 0x7f82540fba80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 >> 172.21.15.189:0/2348881973 conn(0x7f82540f9af0 msgr2=0x7f82540fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 shutdown_connections 2024-04-15T22:52:06.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:06.771+0000 7f825c00a700 1 -- 172.21.15.189:0/2348881973 wait complete. 2024-04-15T22:52:06.768 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:07.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:07 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2348881973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:08.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:08.250 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:08.424 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:08 smithi037 ceph-mon[29191]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:08.447 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:08.495 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 -- 172.21.15.189:0/1984390282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 msgr2=0x7f2ab8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 --2- 172.21.15.189:0/1984390282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8101810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009910 tx=0x7f2aa8009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 -- 172.21.15.189:0/1984390282 shutdown_connections 2024-04-15T22:52:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 --2- 172.21.15.189:0/1984390282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8101810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 -- 172.21.15.189:0/1984390282 >> 172.21.15.189:0/1984390282 conn(0x7f2ab80f9a50 msgr2=0x7f2ab80fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 -- 172.21.15.189:0/1984390282 shutdown_connections 2024-04-15T22:52:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.902+0000 7f2abcc38700 1 -- 172.21.15.189:0/1984390282 wait complete. 2024-04-15T22:52:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2abcc38700 1 Processor -- start 2024-04-15T22:52:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2abcc38700 1 -- start start 2024-04-15T22:52:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2abcc38700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2abcc38700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab8101040 con 0x7f2ab8101400 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2ab659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2ab659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8100b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:55636/0 (socket says 172.21.15.189:55636) 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2ab659c700 1 -- 172.21.15.189:0/1841834260 learned_addr learned my addr 172.21.15.189:0/1841834260 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2ab659c700 1 -- 172.21.15.189:0/1841834260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa80095f0 con 0x7f2ab8101400 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.903+0000 7f2ab659c700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8100b00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8005b40 tx=0x7f2aa8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.904+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa801d070 con 0x7f2ab8101400 2024-04-15T22:52:09.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.904+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab80ff150 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.904+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2aa800be10 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.904+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa800f460 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.904+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab80ff5f0 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.905+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f2aa800b960 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.905+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab8062380 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.905+0000 7f2aa77fe700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f2aa0038490 0x7f2aa003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.905+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2aa804c4b0 con 0x7f2ab8101400 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.905+0000 7f2ab5d9b700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f2aa0038490 0x7f2aa003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:09.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.905+0000 7f2ab5d9b700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f2aa0038490 0x7f2aa003a940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2aac006fd0 tx=0x7f2aac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:09.914+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2aa802a360 con 0x7f2ab8101400 2024-04-15T22:52:10.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.186+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ab818e4e0 con 0x7f2ab8101400 2024-04-15T22:52:10.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.187+0000 7f2aa77fe700 1 -- 172.21.15.189:0/1841834260 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2aa8027030 con 0x7f2ab8101400 2024-04-15T22:52:10.183 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:10.183 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f2aa0038490 msgr2=0x7f2aa003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f2aa0038490 0x7f2aa003a940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2aac006fd0 tx=0x7f2aac006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 msgr2=0x7f2ab8100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8100b00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8005b40 tx=0x7f2aa8004740 comp rx=0 tx=0).stop 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 shutdown_connections 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f2aa0038490 0x7f2aa003a940 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:10.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 --2- 172.21.15.189:0/1841834260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ab8101400 0x7f2ab8100b00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:10.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 >> 172.21.15.189:0/1841834260 conn(0x7f2ab80f9a50 msgr2=0x7f2ab80fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:10.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 shutdown_connections 2024-04-15T22:52:10.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:10.189+0000 7f2abcc38700 1 -- 172.21.15.189:0/1841834260 wait complete. 2024-04-15T22:52:10.186 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:10.529 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:10 smithi037 ceph-mon[29191]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:11.496 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:11 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/1841834260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:11.496 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:11.496 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:11.496 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:11.496 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:11 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:52:11.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:11.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:12.014 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:12.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:12 smithi037 ceph-mon[29191]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:12.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:12 smithi037 ceph-mon[29191]: from='mgr.14162 172.21.15.37:0/1668281737' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:52:12.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:12 smithi037 ceph-mon[29191]: mgrmap e14: smithi037.gtwkck(active, since 78s) 2024-04-15T22:52:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 -- 172.21.15.189:0/1268987160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 msgr2=0x7fcee4100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 --2- 172.21.15.189:0/1268987160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4100230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fced4009910 tx=0x7fced4009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 -- 172.21.15.189:0/1268987160 shutdown_connections 2024-04-15T22:52:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 --2- 172.21.15.189:0/1268987160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4100230 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 -- 172.21.15.189:0/1268987160 >> 172.21.15.189:0/1268987160 conn(0x7fcee40fb3d0 msgr2=0x7fcee40fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 -- 172.21.15.189:0/1268987160 shutdown_connections 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 -- 172.21.15.189:0/1268987160 wait complete. 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 Processor -- start 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.657+0000 7fceebe60700 1 -- start start 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fceebe60700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fceebe60700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4197690 con 0x7fcee40ffe20 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcee9bfc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcee9bfc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:55662/0 (socket says 172.21.15.189:55662) 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcee9bfc700 1 -- 172.21.15.189:0/2967714033 learned_addr learned my addr 172.21.15.189:0/2967714033 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcee9bfc700 1 -- 172.21.15.189:0/2967714033 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced40095f0 con 0x7fcee40ffe20 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcee9bfc700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4197150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fced4005b40 tx=0x7fced4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fced401d070 con 0x7fcee40ffe20 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fced400be10 con 0x7fcee40ffe20 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fced400f460 con 0x7fcee40ffe20 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.658+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee4197890 con 0x7fcee40ffe20 2024-04-15T22:52:13.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.659+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee4197d30 con 0x7fcee40ffe20 2024-04-15T22:52:13.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.659+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 14) v1 ==== 45154+0+0 (secure 0 0 0) 0x7fced40227e0 con 0x7fcee40ffe20 2024-04-15T22:52:13.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.659+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee4190e80 con 0x7fcee40ffe20 2024-04-15T22:52:13.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.660+0000 7fcedaffd700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 0x7fced003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:13.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.660+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fced404c380 con 0x7fcee40ffe20 2024-04-15T22:52:13.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.660+0000 7fcee93fb700 1 -- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 msgr2=0x7fced003a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/712008483 2024-04-15T22:52:13.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.660+0000 7fcee93fb700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 0x7fced003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:52:13.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.664+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fced402b590 con 0x7fcee40ffe20 2024-04-15T22:52:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.861+0000 7fcee93fb700 1 -- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 msgr2=0x7fced003a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/712008483 2024-04-15T22:52:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.861+0000 7fcee93fb700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 0x7fced003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T22:52:13.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.930+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcee402cc30 con 0x7fcee40ffe20 2024-04-15T22:52:13.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.931+0000 7fcedaffd700 1 -- 172.21.15.189:0/2967714033 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fced4027030 con 0x7fcee40ffe20 2024-04-15T22:52:13.926 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:13.927 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 msgr2=0x7fced003a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 0x7fced003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 msgr2=0x7fcee4197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4197150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fced4005b40 tx=0x7fced4004740 comp rx=0 tx=0).stop 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 shutdown_connections 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7fced00384e0 0x7fced003a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 --2- 172.21.15.189:0/2967714033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcee40ffe20 0x7fcee4197150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 >> 172.21.15.189:0/2967714033 conn(0x7fcee40fb3d0 msgr2=0x7fcee40fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 shutdown_connections 2024-04-15T22:52:13.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:13.933+0000 7fceebe60700 1 -- 172.21.15.189:0/2967714033 wait complete. 2024-04-15T22:52:13.930 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:14.069 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:13 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2967714033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:15.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:15.517 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:15.725 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:15.771 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:17.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 -- 172.21.15.189:0/1191633780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 msgr2=0x7f6dc0100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:17.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 --2- 172.21.15.189:0/1191633780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0100230 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009910 tx=0x7f6db4009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:17.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 -- 172.21.15.189:0/1191633780 shutdown_connections 2024-04-15T22:52:17.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 --2- 172.21.15.189:0/1191633780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0100230 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:17.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 -- 172.21.15.189:0/1191633780 >> 172.21.15.189:0/1191633780 conn(0x7f6dc00fb3d0 msgr2=0x7f6dc00fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:17.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 -- 172.21.15.189:0/1191633780 shutdown_connections 2024-04-15T22:52:17.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.201+0000 7f6dc50a1700 1 -- 172.21.15.189:0/1191633780 wait complete. 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dc50a1700 1 Processor -- start 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dc50a1700 1 -- start start 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dc50a1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dc50a1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc01954a0 con 0x7f6dc00ffe20 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dbed9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dbed9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:38966/0 (socket says 172.21.15.189:38966) 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.202+0000 7f6dbed9d700 1 -- 172.21.15.189:0/971771050 learned_addr learned my addr 172.21.15.189:0/971771050 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6dbed9d700 1 -- 172.21.15.189:0/971771050 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db40095f0 con 0x7f6dc00ffe20 2024-04-15T22:52:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6dbed9d700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0194f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6db40092e0 tx=0x7f6db4009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db401d070 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc01956a0 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6db400be10 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db400f460 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.203+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc0195b40 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.204+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 14) v1 ==== 45154+0+0 (secure 0 0 0) 0x7f6db400f6d0 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.204+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc018ea80 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.204+0000 7f6da7fff700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 0x7f6da803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.204+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6db404c460 con 0x7f6dc00ffe20 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.204+0000 7f6dbe59c700 1 -- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 msgr2=0x7f6da803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/712008483 2024-04-15T22:52:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.204+0000 7f6dbe59c700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 0x7f6da803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:52:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.209+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6db4022790 con 0x7f6dc00ffe20 2024-04-15T22:52:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.405+0000 7f6dbe59c700 1 -- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 msgr2=0x7f6da803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/712008483 2024-04-15T22:52:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.405+0000 7f6dbe59c700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 0x7f6da803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-15T22:52:17.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.475+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6dc002cc30 con 0x7f6dc00ffe20 2024-04-15T22:52:17.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.476+0000 7f6da7fff700 1 -- 172.21.15.189:0/971771050 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6db4027070 con 0x7f6dc00ffe20 2024-04-15T22:52:17.472 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:17.472 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 msgr2=0x7f6da803a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 0x7f6da803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 msgr2=0x7f6dc0194f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0194f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6db40092e0 tx=0x7f6db4009e60 comp rx=0 tx=0).stop 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 shutdown_connections 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:6800/712008483,v1:172.21.15.37:6801/712008483] conn(0x7f6da80384e0 0x7f6da803a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 --2- 172.21.15.189:0/971771050 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dc00ffe20 0x7f6dc0194f60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 >> 172.21.15.189:0/971771050 conn(0x7f6dc00fb3d0 msgr2=0x7f6dc00fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 shutdown_connections 2024-04-15T22:52:17.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:17.478+0000 7f6dc50a1700 1 -- 172.21.15.189:0/971771050 wait complete. 2024-04-15T22:52:17.475 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:18.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:18 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/971771050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:19.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:19.120 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:19.328 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:19.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: Active manager daemon smithi037.gtwkck restarted 2024-04-15T22:52:19.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: Activating manager daemon smithi037.gtwkck 2024-04-15T22:52:19.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: osdmap e5: 0 total, 0 up, 0 in 2024-04-15T22:52:19.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: mgrmap e15: smithi037.gtwkck(active, starting, since 0.00323571s) 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr metadata", "who": "smithi037.gtwkck", "id": "smithi037.gtwkck"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: Manager daemon smithi037.gtwkck is now available 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.gtwkck/trash_purge_schedule"}]: dispatch 2024-04-15T22:52:19.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:20.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:20 smithi037 ceph-mon[29191]: [15/Apr/2024:22:52:19] ENGINE Bus STARTING 2024-04-15T22:52:20.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:20 smithi037 ceph-mon[29191]: [15/Apr/2024:22:52:19] ENGINE Serving on https://172.21.15.37:7150 2024-04-15T22:52:20.720 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:20 smithi037 ceph-mon[29191]: [15/Apr/2024:22:52:19] ENGINE Serving on http://172.21.15.37:8765 2024-04-15T22:52:20.720 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:20 smithi037 ceph-mon[29191]: [15/Apr/2024:22:52:19] ENGINE Bus STARTED 2024-04-15T22:52:20.720 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:20 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:20.720 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:20 smithi037 ceph-mon[29191]: mgrmap e16: smithi037.gtwkck(active, since 1.00586s) 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 -- 172.21.15.189:0/1363339929 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 msgr2=0x7f76b80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 --2- 172.21.15.189:0/1363339929 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b80731e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009910 tx=0x7f76b4009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 -- 172.21.15.189:0/1363339929 shutdown_connections 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 --2- 172.21.15.189:0/1363339929 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b80731e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 -- 172.21.15.189:0/1363339929 >> 172.21.15.189:0/1363339929 conn(0x7f76b80fb470 msgr2=0x7f76b80fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 -- 172.21.15.189:0/1363339929 shutdown_connections 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.796+0000 7f76c0109700 1 -- 172.21.15.189:0/1363339929 wait complete. 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.797+0000 7f76c0109700 1 Processor -- start 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.797+0000 7f76c0109700 1 -- start start 2024-04-15T22:52:20.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.797+0000 7f76c0109700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b819b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:20.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.797+0000 7f76c0109700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b819baf0 con 0x7f76b8074ce0 2024-04-15T22:52:20.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76bdea5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b819b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:20.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76bdea5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b819b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:38986/0 (socket says 172.21.15.189:38986) 2024-04-15T22:52:20.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76bdea5700 1 -- 172.21.15.189:0/51583677 learned_addr learned my addr 172.21.15.189:0/51583677 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:20.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76bdea5700 1 -- 172.21.15.189:0/51583677 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40095f0 con 0x7f76b8074ce0 2024-04-15T22:52:20.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76bdea5700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b819b5b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f76b4005950 tx=0x7f76b4009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b401d070 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f76b400be10 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b819bcf0 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.798+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b819c190 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.799+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b400f460 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.799+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 45005+0+0 (secure 0 0 0) 0x7f76b400f690 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.800+0000 7f76aeffd700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f76a4038320 0x7f76a403a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.800+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f76b404c360 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.800+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b81950e0 con 0x7f76b8074ce0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.800+0000 7f76bd6a4700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f76a4038320 0x7f76a403a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:20.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.800+0000 7f76bd6a4700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f76a4038320 0x7f76a403a7d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76a8006fd0 tx=0x7f76a8006e40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:20.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:20.805+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f76b4027030 con 0x7f76b8074ce0 2024-04-15T22:52:21.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.085+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76b802d050 con 0x7f76b8074ce0 2024-04-15T22:52:21.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.086+0000 7f76aeffd700 1 -- 172.21.15.189:0/51583677 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f76b4049030 con 0x7f76b8074ce0 2024-04-15T22:52:21.081 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:21.081 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:21.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f76a4038320 msgr2=0x7f76a403a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f76a4038320 0x7f76a403a7d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76a8006fd0 tx=0x7f76a8006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 msgr2=0x7f76b819b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b819b5b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f76b4005950 tx=0x7f76b4009e60 comp rx=0 tx=0).stop 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 shutdown_connections 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f76a4038320 0x7f76a403a7d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 --2- 172.21.15.189:0/51583677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f76b8074ce0 0x7f76b819b5b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 >> 172.21.15.189:0/51583677 conn(0x7f76b80fb470 msgr2=0x7f76b80fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 shutdown_connections 2024-04-15T22:52:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:21.088+0000 7f76c0109700 1 -- 172.21.15.189:0/51583677 wait complete. 2024-04-15T22:52:21.085 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:22.197 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:22.197 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:21 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/51583677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:22.197 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:21 smithi037 ceph-mon[29191]: mgrmap e17: smithi037.gtwkck(active, since 2s) 2024-04-15T22:52:22.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:22.650 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:23.287 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:23.341 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-15T22:52:23.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:23.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:23.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:23.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:23.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd/host:smithi037", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: Updating smithi037:/etc/ceph/ceph.conf 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: Updating smithi037:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: Updating smithi189:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:25.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:25.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:25.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 -- 172.21.15.189:0/615031550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 msgr2=0x7f1858100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:25.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 --2- 172.21.15.189:0/615031550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858100480 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1854009910 tx=0x7f1854009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:25.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 -- 172.21.15.189:0/615031550 shutdown_connections 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 --2- 172.21.15.189:0/615031550 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858100480 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 -- 172.21.15.189:0/615031550 >> 172.21.15.189:0/615031550 conn(0x7f18580fb970 msgr2=0x7f18580fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 -- 172.21.15.189:0/615031550 shutdown_connections 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.601+0000 7f1860307700 1 -- 172.21.15.189:0/615031550 wait complete. 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.602+0000 7f1860307700 1 Processor -- start 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.602+0000 7f1860307700 1 -- start start 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.602+0000 7f1860307700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.602+0000 7f1860307700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1858197a60 con 0x7f18581000b0 2024-04-15T22:52:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f185e0a3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f185e0a3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:33402/0 (socket says 172.21.15.189:33402) 2024-04-15T22:52:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f185e0a3700 1 -- 172.21.15.189:0/496164763 learned_addr learned my addr 172.21.15.189:0/496164763 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f185e0a3700 1 -- 172.21.15.189:0/496164763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18540095f0 con 0x7f18581000b0 2024-04-15T22:52:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f185e0a3700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858197520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1854005b40 tx=0x7f1854004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f185401d070 con 0x7f18581000b0 2024-04-15T22:52:25.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f18540044d0 con 0x7f18581000b0 2024-04-15T22:52:25.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1858197c60 con 0x7f18581000b0 2024-04-15T22:52:25.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.603+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1854003930 con 0x7f18581000b0 2024-04-15T22:52:25.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.604+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1858198020 con 0x7f18581000b0 2024-04-15T22:52:25.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.605+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f1854003bf0 con 0x7f18581000b0 2024-04-15T22:52:25.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.605+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1858062380 con 0x7f18581000b0 2024-04-15T22:52:25.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.605+0000 7f184f7fe700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1844038580 0x7f184403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:25.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.605+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f185404c4e0 con 0x7f18581000b0 2024-04-15T22:52:25.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.605+0000 7f185d8a2700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1844038580 0x7f184403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:25.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.605+0000 7f185d8a2700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1844038580 0x7f184403aa30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1848006fd0 tx=0x7f1848006e40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:25.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.610+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f185402a950 con 0x7f18581000b0 2024-04-15T22:52:25.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.886+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f185819c010 con 0x7f18581000b0 2024-04-15T22:52:25.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.887+0000 7f184f7fe700 1 -- 172.21.15.189:0/496164763 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f185402a360 con 0x7f18581000b0 2024-04-15T22:52:25.883 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:25.883 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:25.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1844038580 msgr2=0x7f184403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:25.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1844038580 0x7f184403aa30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1848006fd0 tx=0x7f1848006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 msgr2=0x7f1858197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858197520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1854005b40 tx=0x7f1854004c80 comp rx=0 tx=0).stop 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 shutdown_connections 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1844038580 0x7f184403aa30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 --2- 172.21.15.189:0/496164763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f18581000b0 0x7f1858197520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 >> 172.21.15.189:0/496164763 conn(0x7f18580fb970 msgr2=0x7f1858103a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 shutdown_connections 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:25.889+0000 7f1860307700 1 -- 172.21.15.189:0/496164763 wait complete. 2024-04-15T22:52:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: Updating smithi037:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: Updating smithi037:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: Updating smithi189:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-15T22:52:26.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-15T22:52:26.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:26.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: Deploying daemon ceph-exporter.smithi189 on smithi189 2024-04-15T22:52:26.570 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:26 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/496164763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:27.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:27.882 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:28.093 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:29.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.738+0000 7f11923cc700 1 -- 172.21.15.189:0/238870275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 msgr2=0x7f118c100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.738+0000 7f11923cc700 1 --2- 172.21.15.189:0/238870275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c100460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f117c009910 tx=0x7f117c009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 -- 172.21.15.189:0/238870275 shutdown_connections 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 --2- 172.21.15.189:0/238870275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c100460 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 -- 172.21.15.189:0/238870275 >> 172.21.15.189:0/238870275 conn(0x7f118c0fb970 msgr2=0x7f118c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 -- 172.21.15.189:0/238870275 shutdown_connections 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 -- 172.21.15.189:0/238870275 wait complete. 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 Processor -- start 2024-04-15T22:52:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.739+0000 7f11923cc700 1 -- start start 2024-04-15T22:52:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.740+0000 7f11923cc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c1953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.740+0000 7f11923cc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c1958e0 con 0x7f118c100090 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.740+0000 7f118bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c1953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.740+0000 7f118bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c1953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:33412/0 (socket says 172.21.15.189:33412) 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.740+0000 7f118bfff700 1 -- 172.21.15.189:0/3776464835 learned_addr learned my addr 172.21.15.189:0/3776464835 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.740+0000 7f118bfff700 1 -- 172.21.15.189:0/3776464835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0095f0 con 0x7f118c100090 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.741+0000 7f118bfff700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c1953a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f117c005b40 tx=0x7f117c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.741+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f117c01d070 con 0x7f118c100090 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.741+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c195ae0 con 0x7f118c100090 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.741+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f117c0044d0 con 0x7f118c100090 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.741+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f117c003930 con 0x7f118c100090 2024-04-15T22:52:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.741+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c195ea0 con 0x7f118c100090 2024-04-15T22:52:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.742+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f117c003bf0 con 0x7f118c100090 2024-04-15T22:52:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.742+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c062380 con 0x7f118c100090 2024-04-15T22:52:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.742+0000 7f11897fa700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1178038580 0x7f117803aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.742+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f117c04c5c0 con 0x7f118c100090 2024-04-15T22:52:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.742+0000 7f118b7fe700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1178038580 0x7f117803aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.743+0000 7f118b7fe700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1178038580 0x7f117803aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1174006fd0 tx=0x7f1174006e40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:29.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:29.747+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f117c020b60 con 0x7f118c100090 2024-04-15T22:52:30.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.038+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f118c107dc0 con 0x7f118c100090 2024-04-15T22:52:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.039+0000 7f11897fa700 1 -- 172.21.15.189:0/3776464835 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f117c020b60 con 0x7f118c100090 2024-04-15T22:52:30.036 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:30.036 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.041+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1178038580 msgr2=0x7f117803aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.041+0000 7f11923cc700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1178038580 0x7f117803aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1174006fd0 tx=0x7f1174006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.041+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 msgr2=0x7f118c1953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.041+0000 7f11923cc700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c1953a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f117c005b40 tx=0x7f117c004c80 comp rx=0 tx=0).stop 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.042+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 shutdown_connections 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.042+0000 7f11923cc700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f1178038580 0x7f117803aa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.042+0000 7f11923cc700 1 --2- 172.21.15.189:0/3776464835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f118c100090 0x7f118c1953a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:30.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.042+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 >> 172.21.15.189:0/3776464835 conn(0x7f118c0fb970 msgr2=0x7f118c104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:30.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.042+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 shutdown_connections 2024-04-15T22:52:30.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:30.042+0000 7f11923cc700 1 -- 172.21.15.189:0/3776464835 wait complete. 2024-04-15T22:52:30.039 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:31.098 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:31.098 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:31.098 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:31.098 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:31.098 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:52:31.098 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-15T22:52:31.099 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:31.099 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: Deploying daemon crash.smithi189 on smithi189 2024-04-15T22:52:31.099 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:30 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/3776464835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:31.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:31.678 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:34.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 -- 172.21.15.189:0/793481341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 msgr2=0x7f450c0ff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:34.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 --2- 172.21.15.189:0/793481341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c0ff390 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009910 tx=0x7f44fc009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:34.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 -- 172.21.15.189:0/793481341 shutdown_connections 2024-04-15T22:52:34.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 --2- 172.21.15.189:0/793481341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c0ff390 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 -- 172.21.15.189:0/793481341 >> 172.21.15.189:0/793481341 conn(0x7f450c074af0 msgr2=0x7f450c074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 -- 172.21.15.189:0/793481341 shutdown_connections 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.091+0000 7f4510ca1700 1 -- 172.21.15.189:0/793481341 wait complete. 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f4510ca1700 1 Processor -- start 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f4510ca1700 1 -- start start 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f4510ca1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:34.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f4510ca1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c196800 con 0x7f450c0fefc0 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f450a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f450a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c1962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:33442/0 (socket says 172.21.15.189:33442) 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.092+0000 7f450a59c700 1 -- 172.21.15.189:0/1693119273 learned_addr learned my addr 172.21.15.189:0/1693119273 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f450a59c700 1 -- 172.21.15.189:0/1693119273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0095f0 con 0x7f450c0fefc0 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f450a59c700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c1962c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f44fc005b40 tx=0x7f44fc004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44fc01d070 con 0x7f450c0fefc0 2024-04-15T22:52:34.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f44fc0044d0 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c196a00 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44fc003930 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.093+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c196dc0 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.094+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f44fc003ba0 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.094+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c107240 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.094+0000 7f44fb7fe700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f44f4038620 0x7f44f403aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.094+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f44fc04c710 con 0x7f450c0fefc0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.095+0000 7f4509d9b700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f44f4038620 0x7f44f403aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:34.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.095+0000 7f4509d9b700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f44f4038620 0x7f44f403aad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4500006fd0 tx=0x7f4500006e40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:34.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.100+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f44fc02a950 con 0x7f450c0fefc0 2024-04-15T22:52:34.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.393+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f450c068540 con 0x7f450c0fefc0 2024-04-15T22:52:34.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.394+0000 7f44fb7fe700 1 -- 172.21.15.189:0/1693119273 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f44fc02a360 con 0x7f450c0fefc0 2024-04-15T22:52:34.391 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:34.391 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f44f4038620 msgr2=0x7f44f403aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f44f4038620 0x7f44f403aad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4500006fd0 tx=0x7f4500006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 msgr2=0x7f450c1962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c1962c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f44fc005b40 tx=0x7f44fc004c80 comp rx=0 tx=0).stop 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 shutdown_connections 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f44f4038620 0x7f44f403aad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 --2- 172.21.15.189:0/1693119273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f450c0fefc0 0x7f450c1962c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 >> 172.21.15.189:0/1693119273 conn(0x7f450c074af0 msgr2=0x7f450c1031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 shutdown_connections 2024-04-15T22:52:34.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:34.397+0000 7f4510ca1700 1 -- 172.21.15.189:0/1693119273 wait complete. 2024-04-15T22:52:34.395 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:34.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:34.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:52:34.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:34.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:34.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:34.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:35.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:35 smithi037 ceph-mon[29191]: Deploying daemon node-exporter.smithi189 on smithi189 2024-04-15T22:52:35.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:35 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/1693119273' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:36.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:36.099 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:36.314 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:39.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.160+0000 7f056e270700 1 -- 172.21.15.189:0/1718444742 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 msgr2=0x7f0568100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:39.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.160+0000 7f056e270700 1 --2- 172.21.15.189:0/1718444742 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568100480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0550009910 tx=0x7f0550009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:39.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.160+0000 7f056e270700 1 -- 172.21.15.189:0/1718444742 shutdown_connections 2024-04-15T22:52:39.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.160+0000 7f056e270700 1 --2- 172.21.15.189:0/1718444742 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568100480 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:39.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.160+0000 7f056e270700 1 -- 172.21.15.189:0/1718444742 >> 172.21.15.189:0/1718444742 conn(0x7f05680fb970 msgr2=0x7f05680fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:39.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.161+0000 7f056e270700 1 -- 172.21.15.189:0/1718444742 shutdown_connections 2024-04-15T22:52:39.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.161+0000 7f056e270700 1 -- 172.21.15.189:0/1718444742 wait complete. 2024-04-15T22:52:39.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.161+0000 7f056e270700 1 Processor -- start 2024-04-15T22:52:39.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.161+0000 7f056e270700 1 -- start start 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f056e270700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f056e270700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0568195860 con 0x7f05681000b0 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f0567fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f0567fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:38370/0 (socket says 172.21.15.189:38370) 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f0567fff700 1 -- 172.21.15.189:0/2952352441 learned_addr learned my addr 172.21.15.189:0/2952352441 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f0567fff700 1 -- 172.21.15.189:0/2952352441 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500095f0 con 0x7f05681000b0 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.162+0000 7f0567fff700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568195320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0550005b40 tx=0x7f0550004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.163+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f055001d070 con 0x7f05681000b0 2024-04-15T22:52:39.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.163+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0568195a60 con 0x7f05681000b0 2024-04-15T22:52:39.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.163+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f05500044d0 con 0x7f05681000b0 2024-04-15T22:52:39.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.163+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0550003930 con 0x7f05681000b0 2024-04-15T22:52:39.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.163+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0568195e20 con 0x7f05681000b0 2024-04-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.164+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0568062380 con 0x7f05681000b0 2024-04-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.164+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f0550003c30 con 0x7f05681000b0 2024-04-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.164+0000 7f05657fa700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0554038580 0x7f055403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.164+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f055004c3e0 con 0x7f05681000b0 2024-04-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.164+0000 7f05677fe700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0554038580 0x7f055403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.165+0000 7f05677fe700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0554038580 0x7f055403aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0558006fd0 tx=0x7f0558006e40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.169+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f055002a540 con 0x7f05681000b0 2024-04-15T22:52:39.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.449+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0568107e50 con 0x7f05681000b0 2024-04-15T22:52:39.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.449+0000 7f05657fa700 1 -- 172.21.15.189:0/2952352441 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f055002a360 con 0x7f05681000b0 2024-04-15T22:52:39.445 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:39.446 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:39.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0554038580 msgr2=0x7f055403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0554038580 0x7f055403aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0558006fd0 tx=0x7f0558006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 msgr2=0x7f0568195320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568195320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0550005b40 tx=0x7f0550004c80 comp rx=0 tx=0).stop 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 shutdown_connections 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0554038580 0x7f055403aa30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 --2- 172.21.15.189:0/2952352441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05681000b0 0x7f0568195320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 >> 172.21.15.189:0/2952352441 conn(0x7f05680fb970 msgr2=0x7f0568104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 shutdown_connections 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:39.451+0000 7f056e270700 1 -- 172.21.15.189:0/2952352441 wait complete. 2024-04-15T22:52:39.448 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.lanqnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.lanqnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T22:52:39.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:52:39.820 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:40.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:40 smithi037 ceph-mon[29191]: Deploying daemon mgr.smithi189.lanqnp on smithi189 2024-04-15T22:52:40.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:40 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2952352441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:41.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:41.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:41.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:41 smithi037 ceph-mon[29191]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:41.892 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: Deploying daemon mon.smithi189 on smithi189 2024-04-15T22:52:43.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:42 smithi037 ceph-mon[29191]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:44.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:44 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 -- 172.21.15.189:0/3409977667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 msgr2=0x7fc15c100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 --2- 172.21.15.189:0/3409977667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c100480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009910 tx=0x7fc14c009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 -- 172.21.15.189:0/3409977667 shutdown_connections 2024-04-15T22:52:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 --2- 172.21.15.189:0/3409977667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c100480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 -- 172.21.15.189:0/3409977667 >> 172.21.15.189:0/3409977667 conn(0x7fc15c0fb970 msgr2=0x7fc15c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 -- 172.21.15.189:0/3409977667 shutdown_connections 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 -- 172.21.15.189:0/3409977667 wait complete. 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 Processor -- start 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.115+0000 7fc16166c700 1 -- start start 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.116+0000 7fc16166c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.116+0000 7fc16166c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c195860 con 0x7fc15c1000b0 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.116+0000 7fc15affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.116+0000 7fc15affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.189:33884/0 (socket says 172.21.15.189:33884) 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.116+0000 7fc15affd700 1 -- 172.21.15.189:0/2518844162 learned_addr learned my addr 172.21.15.189:0/2518844162 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.116+0000 7fc15affd700 1 -- 172.21.15.189:0/2518844162 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0095f0 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.117+0000 7fc15affd700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c195320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc14c005b40 tx=0x7fc14c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.117+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14c01d070 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.117+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc14c0044d0 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.117+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c195a60 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.117+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14c003930 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.117+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c195e20 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.118+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c062380 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.118+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fc14c003bf0 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.118+0000 7fc143fff700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fc144038580 0x7fc14403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.118+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc14c04c570 con 0x7fc15c1000b0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.118+0000 7fc15a7fc700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fc144038580 0x7fc14403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:45.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.119+0000 7fc15a7fc700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fc144038580 0x7fc14403aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc150006fd0 tx=0x7fc150006e40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.123+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc14c020b60 con 0x7fc15c1000b0 2024-04-15T22:52:45.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.400+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc15c107e10 con 0x7fc15c1000b0 2024-04-15T22:52:45.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.400+0000 7fc143fff700 1 -- 172.21.15.189:0/2518844162 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc14c020b60 con 0x7fc15c1000b0 2024-04-15T22:52:45.396 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:45.397 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:49:44.850697Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:52:45.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.402+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fc144038580 msgr2=0x7fc14403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.402+0000 7fc16166c700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fc144038580 0x7fc14403aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc150006fd0 tx=0x7fc150006e40 comp rx=0 tx=0).stop 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.402+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 msgr2=0x7fc15c195320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.402+0000 7fc16166c700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c195320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc14c005b40 tx=0x7fc14c004c80 comp rx=0 tx=0).stop 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.403+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 shutdown_connections 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.403+0000 7fc16166c700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fc144038580 0x7fc14403aa30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.403+0000 7fc16166c700 1 --2- 172.21.15.189:0/2518844162 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc15c1000b0 0x7fc15c195320 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.403+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 >> 172.21.15.189:0/2518844162 conn(0x7fc15c0fb970 msgr2=0x7fc15c104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.403+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 shutdown_connections 2024-04-15T22:52:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:45.403+0000 7fc16166c700 1 -- 172.21.15.189:0/2518844162 wait complete. 2024-04-15T22:52:45.400 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-15T22:52:45.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:45 smithi037 ceph-mon[29191]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:46.567 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 podman[36157]: 2024-04-15 22:52:46.537611362 +0000 UTC m=+0.131416093 container create e2331fdccbaadcecc0ed7d7164a15fae3fefd16847d60736a97dedfe499903fd (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-04-15T22:52:46.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:46 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2518844162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 podman[36157]: 2024-04-15 22:52:46.781831262 +0000 UTC m=+0.375635992 container init e2331fdccbaadcecc0ed7d7164a15fae3fefd16847d60736a97dedfe499903fd (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 podman[36157]: 2024-04-15 22:52:46.785197072 +0000 UTC m=+0.379001804 container start e2331fdccbaadcecc0ed7d7164a15fae3fefd16847d60736a97dedfe499903fd (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189, io.buildah.version=1.29.1, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD) 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mon, pid 2 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: pidfile_write: ignore empty --pid-file 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: load: jerasure load: lrc 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: RocksDB version: 7.9.2 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Git sha 0 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Compile date 2023-08-03 19:21:13 2024-04-15T22:52:46.833 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: DB SUMMARY 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: DB Session ID: HDZ9KJPO6CX2KXRASOBP 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: CURRENT file: CURRENT 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi189/store.db dir, Total Num: 0, files: 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi189/store.db: 000004.log size: 511 ; 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.error_if_exists: 0 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.create_if_missing: 0 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.paranoid_checks: 1 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.env: 0x55c7250cb720 2024-04-15T22:52:46.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.fs: PosixFileSystem 2024-04-15T22:52:46.835 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.info_log: 0x55c726c372c0 2024-04-15T22:52:46.835 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T22:52:46.835 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.statistics: (nil) 2024-04-15T22:52:46.835 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.use_fsync: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_log_file_size: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.allow_fallocate: 1 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.use_direct_reads: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.db_log_dir: 2024-04-15T22:52:46.836 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.wal_dir: 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.write_buffer_manager: 0x55c725ea83c0 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T22:52:46.837 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.unordered_write: 0 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.row_cache: None 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.wal_filter: None 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T22:52:46.838 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.two_write_queues: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.wal_compression: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.atomic_flush: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.log_readahead_size: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-15T22:52:46.839 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_background_jobs: 2 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_background_compactions: -1 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_subcompactions: 1 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_open_files: -1 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T22:52:46.840 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_background_flushes: -1 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Compression algorithms supported: 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kZSTD supported: 0 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kXpressCompression supported: 0 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kZlibCompression supported: 1 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kSnappyCompression supported: 1 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kLZ4Compression supported: 1 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: kBZip2Compression supported: 0 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi189/store.db/MANIFEST-000005 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T22:52:46.841 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.merge_operator: 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_filter: None 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c726c373c0) 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: index_type: 0 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: data_block_index_type: 0 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: index_shortening: 1 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T22:52:46.842 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: checksum: 4 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: no_block_cache: 0 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache: 0x55c725f491f0 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache_options: 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: capacity : 536870912 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: num_shard_bits : 4 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: strict_capacity_limit : 0 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache_compressed: (nil) 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: persistent_cache: (nil) 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_size: 4096 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_size_deviation: 10 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_restart_interval: 16 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: index_block_restart_interval: 1 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: metadata_block_size: 4096 2024-04-15T22:52:46.843 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: partition_filters: 0 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: use_delta_encoding: 1 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: filter_policy: bloomfilter 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: whole_key_filtering: 1 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: verify_compression: 0 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: format_version: 5 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: enable_index_compression: 1 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_align: 0 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: max_auto_readahead_size: 262144 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: prepopulate_block_cache: 0 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: initial_auto_readahead_size: 8192 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression: NoCompression 2024-04-15T22:52:46.844 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.num_levels: 7 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-15T22:52:46.845 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T22:52:46.846 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T22:52:46.847 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T22:52:46.847 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T22:52:46.847 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T22:52:46.847 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.arena_block_size: 1048576 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T22:52:46.848 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.table_properties_collectors: 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.inplace_update_support: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.bloom_locality: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.max_successive_merges: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T22:52:46.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.ttl: 2592000 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.enable_blob_files: false 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.min_blob_size: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-15T22:52:46.850 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.blob_file_starting_level: 0 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi189/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b7817336-ce11-4df3-b8e1-6ccd49bf4529 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713221566815912, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713221566816375, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713221566, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b7817336-ce11-4df3-b8e1-6ccd49bf4529", "db_session_id": "HDZ9KJPO6CX2KXRASOBP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713221566816454, "job": 1, "event": "recovery_finished"} 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi189/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c725fe6000 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: DB pointer 0x55c725fd2000 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** DB Stats ** 2024-04-15T22:52:46.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** Compaction Stats [default] ** 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** Compaction Stats [default] ** 2024-04-15T22:52:46.852 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Block cache BinnedLRUCache@0x55c725f491f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-15T22:52:46.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189 does not exist in monmap, will attempt to join an existing cluster 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: using public_addr v2:172.21.15.189:0/0 -> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: starting mon.smithi189 rank -1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi189 fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(???) e0 preinit fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).mds e1 new map 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).mds e1 print_map 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: e1 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: legacy client fscid: -1 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: No filesystems configured 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-15T22:52:46.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: Deploying daemon mgr.smithi189.lanqnp on smithi189 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/2952352441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:46.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: Deploying daemon mon.smithi189 on smithi189 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/2518844162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: mon.smithi189@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expand map: {default=false} 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta from 'false' to 'false' 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expanded map: {default=false} 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expand map: {default=info} 2024-04-15T22:52:46.856 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta from 'info' to 'info' 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expanded map: {default=info} 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expand map: {default=daemon} 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expanded map: {default=daemon} 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expand map: {default=debug} 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta from 'debug' to 'debug' 2024-04-15T22:52:46.857 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 ceph-mon[36179]: expand_channel_meta expanded map: {default=debug} 2024-04-15T22:52:47.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 bash[36157]: e2331fdccbaadcecc0ed7d7164a15fae3fefd16847d60736a97dedfe499903fd 2024-04-15T22:52:47.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:46 smithi189 systemd[1]: Started Ceph mon.smithi189 for 60ff499a-fb7a-11ee-bc8f-c7b262605968. 2024-04-15T22:52:47.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-15T22:52:47.314 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:52:47.524 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:52:52.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 -- 172.21.15.189:0/1668488494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00540035c0 msgr2=0x7f0054005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:52.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 --2- 172.21.15.189:0/1668488494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00540035c0 0x7f0054005a40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f00680ffea0 tx=0x7f006400feb0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 -- 172.21.15.189:0/1668488494 shutdown_connections 2024-04-15T22:52:52.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 --2- 172.21.15.189:0/1668488494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00540035c0 0x7f0054005a40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 -- 172.21.15.189:0/1668488494 >> 172.21.15.189:0/1668488494 conn(0x7f00680fba50 msgr2=0x7f00680fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:52.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 -- 172.21.15.189:0/1668488494 shutdown_connections 2024-04-15T22:52:52.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.099+0000 7f006f472700 1 -- 172.21.15.189:0/1668488494 wait complete. 2024-04-15T22:52:52.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006f472700 1 Processor -- start 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006f472700 1 -- start start 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006f472700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006f472700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0068195b40 0x7f0068199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006f472700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068196150 con 0x7f0068195b40 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006f472700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00681962c0 con 0x7f00540035c0 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.100+0000 7f006d20e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006d20e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60636/0 (socket says 172.21.15.189:60636) 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006d20e700 1 -- 172.21.15.189:0/712901108 learned_addr learned my addr 172.21.15.189:0/712901108 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006d20e700 1 -- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 msgr2=0x7f0068195600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006d20e700 1 -- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 msgr2=0x7f0068195600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006d20e700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T22:52:52.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006d20e700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006ca0d700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0068195b40 0x7f0068199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006ca0d700 1 -- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 msgr2=0x7f0068195600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006ca0d700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006ca0d700 1 -- 172.21.15.189:0/712901108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006400f900 con 0x7f0068195b40 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f006ca0d700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0068195b40 0x7f0068199fb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f005800b500 tx=0x7f005800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.101+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0058011800 con 0x7f0068195b40 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.102+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0058011e10 con 0x7f0068195b40 2024-04-15T22:52:52.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.102+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006819a5b0 con 0x7f0068195b40 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.102+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f005800f2e0 con 0x7f0068195b40 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.102+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006819ab00 con 0x7f0068195b40 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.103+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006819ae30 con 0x7f0068195b40 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.103+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f0058011960 con 0x7f0068195b40 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.104+0000 7f005e7fc700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f004c06c750 0x7f004c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.104+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f0058089d00 con 0x7f0068195b40 2024-04-15T22:52:52.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.104+0000 7f006d20e700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f004c06c750 0x7f004c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:52.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.104+0000 7f006d20e700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f004c06c750 0x7f004c06ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0064001eb0 tx=0x7f0064005960 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:52.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.109+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f00580554a0 con 0x7f0068195b40 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: mon.smithi037 calling monitor election 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.lanqnp/crt"}]: dispatch 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: mon.smithi189 calling monitor election 2024-04-15T22:52:52.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: mon.smithi037 is new leader, mons smithi037,smithi189 in quorum (ranks 0,1) 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: monmap e2: 2 mons at {smithi037=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],smithi189=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: fsmap 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: osdmap e5: 0 total, 0 up, 0 in 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: mgrmap e17: smithi037.gtwkck(active, since 33s) 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: Standby manager daemon smithi189.lanqnp started 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: overall HEALTH_OK 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.lanqnp/key"}]: dispatch 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:52:52.217 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:52:52.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: mon.smithi037 calling monitor election 2024-04-15T22:52:52.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:52.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.lanqnp/crt"}]: dispatch 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: mon.smithi189 calling monitor election 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: mon.smithi037 is new leader, mons smithi037,smithi189 in quorum (ranks 0,1) 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: monmap e2: 2 mons at {smithi037=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0],smithi189=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: fsmap 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: osdmap e5: 0 total, 0 up, 0 in 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: mgrmap e17: smithi037.gtwkck(active, since 33s) 2024-04-15T22:52:52.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: Standby manager daemon smithi189.lanqnp started 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: overall HEALTH_OK 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.lanqnp/key"}]: dispatch 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.? 172.21.15.189:0/1063028025' entity='mgr.smithi189.lanqnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:52.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.389+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00681a0250 con 0x7f0068195b40 2024-04-15T22:52:52.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.389+0000 7f005e7fc700 1 -- 172.21.15.189:0/712901108 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7f0058058ac0 con 0x7f0068195b40 2024-04-15T22:52:52.384 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-15T22:52:52.384 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:52:46.843624Z","created":"2024-04-15T22:49:44.850697Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi189","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T22:52:52.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.391+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f004c06c750 msgr2=0x7f004c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.391+0000 7f006f472700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f004c06c750 0x7f004c06ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0064001eb0 tx=0x7f0064005960 comp rx=0 tx=0).stop 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0068195b40 msgr2=0x7f0068199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0068195b40 0x7f0068199fb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f005800b500 tx=0x7f005800b8c0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 shutdown_connections 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f004c06c750 0x7f004c06ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00540035c0 0x7f0068195600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 --2- 172.21.15.189:0/712901108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0068195b40 0x7f0068199fb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 >> 172.21.15.189:0/712901108 conn(0x7f00680fba50 msgr2=0x7f006818d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 shutdown_connections 2024-04-15T22:52:52.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:52:52.392+0000 7f006f472700 1 -- 172.21.15.189:0/712901108 wait complete. 2024-04-15T22:52:52.388 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-04-15T22:52:52.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:52 smithi189 ceph-mon[36179]: mgrmap e18: smithi037.gtwkck(active, since 33s), standbys: smithi189.lanqnp 2024-04-15T22:52:52.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr metadata", "who": "smithi189.lanqnp", "id": "smithi189.lanqnp"}]: dispatch 2024-04-15T22:52:52.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:52:52.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:52 smithi189 ceph-mon[36179]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:52.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:52 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/712901108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:52.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:53.018 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-15T22:52:53.018 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-15T22:52:53.243 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:52:53.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:52 smithi037 ceph-mon[29191]: mgrmap e18: smithi037.gtwkck(active, since 33s), standbys: smithi189.lanqnp 2024-04-15T22:52:53.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr metadata", "who": "smithi189.lanqnp", "id": "smithi189.lanqnp"}]: dispatch 2024-04-15T22:52:53.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:52:53.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:52 smithi037 ceph-mon[29191]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:53.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:52 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/712901108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:52:53.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:52:54.683 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:54 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:54.782 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:54 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.042+0000 7f8c7b949700 1 -- 172.21.15.37:0/3702307357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 msgr2=0x7f8c740ff360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.042+0000 7f8c7b949700 1 --2- 172.21.15.37:0/3702307357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c740ff360 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009910 tx=0x7f8c70009c20 comp rx=0 tx=0).stop 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.042+0000 7f8c7b949700 1 -- 172.21.15.37:0/3702307357 shutdown_connections 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.043+0000 7f8c7b949700 1 --2- 172.21.15.37:0/3702307357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c740ff360 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.043+0000 7f8c7b949700 1 -- 172.21.15.37:0/3702307357 >> 172.21.15.37:0/3702307357 conn(0x7f8c740fa850 msgr2=0x7f8c740fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.043+0000 7f8c7b949700 1 -- 172.21.15.37:0/3702307357 shutdown_connections 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.043+0000 7f8c7b949700 1 -- 172.21.15.37:0/3702307357 wait complete. 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.043+0000 7f8c7b949700 1 Processor -- start 2024-04-15T22:52:55.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.043+0000 7f8c7b949700 1 -- start start 2024-04-15T22:52:55.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c7b949700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c741940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:55.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c7b949700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c74194630 0x7f8c74194aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c7b949700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740697b0 con 0x7f8c740fef90 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c7b949700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741a1760 con 0x7f8c74194630 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c796e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c741940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c796e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c741940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55290/0 (socket says 172.21.15.37:55290) 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c796e5700 1 -- 172.21.15.37:0/2183877223 learned_addr learned my addr 172.21.15.37:0/2183877223 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c78ee4700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c74194630 0x7f8c74194aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c796e5700 1 -- 172.21.15.37:0/2183877223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c74194630 msgr2=0x7f8c74194aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c796e5700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c74194630 0x7f8c74194aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c796e5700 1 -- 172.21.15.37:0/2183877223 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c700095f0 con 0x7f8c740fef90 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.044+0000 7f8c78ee4700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c74194630 0x7f8c74194aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.045+0000 7f8c796e5700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c741940f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8c70005b40 tx=0x7f8c7000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.045+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c7001d070 con 0x7f8c740fef90 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.045+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8c70005140 con 0x7f8c740fef90 2024-04-15T22:52:55.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.045+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c741a1900 con 0x7f8c740fef90 2024-04-15T22:52:55.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.045+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c70017440 con 0x7f8c740fef90 2024-04-15T22:52:55.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.045+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c741a1dc0 con 0x7f8c740fef90 2024-04-15T22:52:55.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.046+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58005320 con 0x7f8c740fef90 2024-04-15T22:52:55.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.047+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f8c70017700 con 0x7f8c740fef90 2024-04-15T22:52:55.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.047+0000 7f8c667fc700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8c6006c760 0x7f8c6006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:52:55.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.047+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8c7008bc60 con 0x7f8c740fef90 2024-04-15T22:52:55.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.047+0000 7f8c78ee4700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8c6006c760 0x7f8c6006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:52:55.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.048+0000 7f8c78ee4700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8c6006c760 0x7f8c6006ec10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c680079b0 tx=0x7f8c68008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:52:55.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.052+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8c70057420 con 0x7f8c740fef90 2024-04-15T22:52:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.262+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8c58005190 con 0x7f8c740fef90 2024-04-15T22:52:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.263+0000 7f8c667fc700 1 -- 172.21.15.37:0/2183877223 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f8c70026090 con 0x7f8c740fef90 2024-04-15T22:52:55.258 INFO:teuthology.orchestra.run.smithi037.stdout:# minimal ceph.conf for 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:52:55.258 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-04-15T22:52:55.258 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = 60ff499a-fb7a-11ee-bc8f-c7b262605968 2024-04-15T22:52:55.258 INFO:teuthology.orchestra.run.smithi037.stdout: mon_host = [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-15T22:52:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8c6006c760 msgr2=0x7f8c6006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8c6006c760 0x7f8c6006ec10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c680079b0 tx=0x7f8c68008040 comp rx=0 tx=0).stop 2024-04-15T22:52:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 msgr2=0x7f8c741940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c741940f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8c70005b40 tx=0x7f8c7000bda0 comp rx=0 tx=0).stop 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 shutdown_connections 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8c6006c760 0x7f8c6006ec10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c740fef90 0x7f8c741940f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 --2- 172.21.15.37:0/2183877223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c74194630 0x7f8c74194aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 >> 172.21.15.37:0/2183877223 conn(0x7f8c740fa850 msgr2=0x7f8c74104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 shutdown_connections 2024-04-15T22:52:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:52:55.265+0000 7f8c7b949700 1 -- 172.21.15.37:0/2183877223 wait complete. 2024-04-15T22:52:55.587 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:55.587 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:55.588 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:55 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2183877223' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:55.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:55 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2183877223' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:55.893 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-15T22:52:55.894 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:52:55.894 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T22:52:55.924 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:52:55.924 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:52:55.992 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:52:55.992 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T22:52:56.024 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:52:56.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:52:56.094 INFO:tasks.cephadm:Deploying OSDs... 2024-04-15T22:52:56.094 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:52:56.094 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T22:52:56.110 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T22:52:56.110 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-15 22:52:09.799726576 +0000 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-15 22:46:34.908219293 +0000 2024-04-15T22:52:56.166 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-15 22:46:34.908219293 +0000 2024-04-15T22:52:56.167 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-04-15 22:46:34.908219293 +0000 2024-04-15T22:52:56.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T22:52:56.233 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-15T22:52:56.233 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-15T22:52:56.233 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000205653 s, 2.5 MB/s 2024-04-15T22:52:56.235 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T22:52:56.292 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-15 22:52:09.799726576 +0000 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-15 22:46:35.258214824 +0000 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-15 22:46:35.258214824 +0000 2024-04-15T22:52:56.349 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-04-15 22:46:35.258214824 +0000 2024-04-15T22:52:56.349 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T22:52:56.417 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-15T22:52:56.417 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-15T22:52:56.417 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000197607 s, 2.6 MB/s 2024-04-15T22:52:56.418 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T22:52:56.474 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-04-15T22:52:56.528 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-15 22:52:09.799726576 +0000 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-15 22:46:35.622210175 +0000 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-15 22:46:35.622210175 +0000 2024-04-15T22:52:56.529 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-04-15 22:46:35.622210175 +0000 2024-04-15T22:52:56.529 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T22:52:56.595 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-15T22:52:56.595 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-15T22:52:56.595 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000216379 s, 2.4 MB/s 2024-04-15T22:52:56.597 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T22:52:56.653 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-04-15T22:52:56.677 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:56 smithi037 ceph-mon[29191]: Updating smithi037:/etc/ceph/ceph.conf 2024-04-15T22:52:56.677 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:56 smithi037 ceph-mon[29191]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-15T22:52:56.677 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:56 smithi037 ceph-mon[29191]: Updating smithi189:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:56.678 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:56 smithi037 ceph-mon[29191]: Updating smithi037:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:56.678 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:56 smithi037 ceph-mon[29191]: Reconfiguring mon.smithi037 (unknown last config time)... 2024-04-15T22:52:56.678 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:56 smithi037 ceph-mon[29191]: Reconfiguring daemon mon.smithi037 on smithi037 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-15 22:52:09.800726564 +0000 2024-04-15T22:52:56.692 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-15 22:46:36.196202845 +0000 2024-04-15T22:52:56.693 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-15 22:46:36.196202845 +0000 2024-04-15T22:52:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-04-15 22:46:36.196202845 +0000 2024-04-15T22:52:56.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T22:52:56.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:56 smithi189 ceph-mon[36179]: Updating smithi037:/etc/ceph/ceph.conf 2024-04-15T22:52:56.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:56 smithi189 ceph-mon[36179]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-15T22:52:56.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:56 smithi189 ceph-mon[36179]: Updating smithi189:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:56.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:56 smithi189 ceph-mon[36179]: Updating smithi037:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T22:52:56.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:56 smithi189 ceph-mon[36179]: Reconfiguring mon.smithi037 (unknown last config time)... 2024-04-15T22:52:56.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:56 smithi189 ceph-mon[36179]: Reconfiguring daemon mon.smithi037 on smithi037 2024-04-15T22:52:56.759 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-15T22:52:56.759 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-15T22:52:56.759 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000184742 s, 2.8 MB/s 2024-04-15T22:52:56.760 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T22:52:56.817 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:52:56.817 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T22:52:56.835 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T22:52:56.835 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-04-15T22:52:56.892 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-15 22:52:45.887607590 +0000 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-15 22:46:34.902904395 +0000 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-15 22:46:34.902904395 +0000 2024-04-15T22:52:56.893 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-15 22:46:34.902904395 +0000 2024-04-15T22:52:56.893 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T22:52:56.962 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-15T22:52:56.962 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-15T22:52:56.962 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000232822 s, 2.2 MB/s 2024-04-15T22:52:56.964 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T22:52:57.022 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-15 22:52:45.888607571 +0000 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-15 22:46:35.279897179 +0000 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-15 22:46:35.279897179 +0000 2024-04-15T22:52:57.079 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-15 22:46:35.279897179 +0000 2024-04-15T22:52:57.079 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T22:52:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-15T22:52:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-15T22:52:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000194826 s, 2.6 MB/s 2024-04-15T22:52:57.148 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T22:52:57.205 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-04-15T22:52:57.261 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T22:52:57.261 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:57.261 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-15T22:52:57.262 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:57.262 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:57.262 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-15 22:52:45.888607571 +0000 2024-04-15T22:52:57.262 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-15 22:46:35.648890117 +0000 2024-04-15T22:52:57.262 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-15 22:46:35.648890117 +0000 2024-04-15T22:52:57.262 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-15 22:46:35.648890117 +0000 2024-04-15T22:52:57.262 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T22:52:57.328 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-15T22:52:57.328 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-15T22:52:57.328 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000196944 s, 2.6 MB/s 2024-04-15T22:52:57.329 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T22:52:57.386 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-15 22:52:45.889607553 +0000 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-15 22:46:36.003883322 +0000 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-15 22:46:36.003883322 +0000 2024-04-15T22:52:57.441 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-15 22:46:36.003883322 +0000 2024-04-15T22:52:57.441 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T22:52:57.508 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-15T22:52:57.508 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-15T22:52:57.508 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000210998 s, 2.4 MB/s 2024-04-15T22:52:57.509 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T22:52:57.565 INFO:tasks.cephadm:Deploying osd.0 on smithi037 with /dev/vg_nvme/lv_4... 2024-04-15T22:52:57.566 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T22:52:57.714 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:57 smithi037 ceph-mon[29191]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:57.714 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:57 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:57.714 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:57 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:57 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi037.gtwkck", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:57 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:57 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:57 smithi189 ceph-mon[36179]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:57 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:57 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:57 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi037.gtwkck", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:57 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:52:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:57 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:58.281 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:52:58.542 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:58 smithi037 ceph-mon[29191]: Reconfiguring mgr.smithi037.gtwkck (unknown last config time)... 2024-04-15T22:52:58.542 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:58 smithi037 ceph-mon[29191]: Reconfiguring daemon mgr.smithi037.gtwkck on smithi037 2024-04-15T22:52:58.542 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:58.542 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:58.542 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi037", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-15T22:52:58.543 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:58.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:58 smithi189 ceph-mon[36179]: Reconfiguring mgr.smithi037.gtwkck (unknown last config time)... 2024-04-15T22:52:58.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:58 smithi189 ceph-mon[36179]: Reconfiguring daemon mgr.smithi037.gtwkck on smithi037 2024-04-15T22:52:58.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:58.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:52:58.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi037", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-15T22:52:58.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:52:59.709 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:59 smithi037 ceph-mon[29191]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:59.709 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:59 smithi037 ceph-mon[29191]: Reconfiguring ceph-exporter.smithi037 (monmap changed)... 2024-04-15T22:52:59.709 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:52:59 smithi037 ceph-mon[29191]: Reconfiguring daemon ceph-exporter.smithi037 on smithi037 2024-04-15T22:52:59.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:59 smithi189 ceph-mon[36179]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:52:59.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:59 smithi189 ceph-mon[36179]: Reconfiguring ceph-exporter.smithi037 (monmap changed)... 2024-04-15T22:52:59.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:52:59 smithi189 ceph-mon[36179]: Reconfiguring daemon ceph-exporter.smithi037 on smithi037 2024-04-15T22:53:01.163 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T22:53:01.163 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T22:53:01.163 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-04-15T22:53:01.163 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-04-15T22:53:01.164 INFO:teuthology.orchestra.run.smithi037.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111364 s, 942 MB/s 2024-04-15T22:53:01.164 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-04-15T22:53:01.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:01.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:01.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:01.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: Reconfiguring crash.smithi037 (monmap changed)... 2024-04-15T22:53:01.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi037", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:53:01.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:01.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:01 smithi037 ceph-mon[29191]: Reconfiguring daemon crash.smithi037 on smithi037 2024-04-15T22:53:01.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:01.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:01.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:01.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: Reconfiguring crash.smithi037 (monmap changed)... 2024-04-15T22:53:01.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi037", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:53:01.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:01.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:01 smithi189 ceph-mon[36179]: Reconfiguring daemon crash.smithi037 on smithi037 2024-04-15T22:53:02.525 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi037:vg_nvme/lv_4 2024-04-15T22:53:03.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:03.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:03.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:03 smithi037 ceph-mon[29191]: Reconfiguring alertmanager.smithi037 (dependencies changed)... 2024-04-15T22:53:03.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:03 smithi037 ceph-mon[29191]: Reconfiguring daemon alertmanager.smithi037 on smithi037 2024-04-15T22:53:03.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:03 smithi037 ceph-mon[29191]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:03.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:03.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:03.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:03 smithi189 ceph-mon[36179]: Reconfiguring alertmanager.smithi037 (dependencies changed)... 2024-04-15T22:53:03.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:03 smithi189 ceph-mon[36179]: Reconfiguring daemon alertmanager.smithi037 on smithi037 2024-04-15T22:53:03.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:03 smithi189 ceph-mon[36179]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:53:04.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:04.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:05.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:05 smithi189 ceph-mon[36179]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:05.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:05 smithi037 ceph-mon[29191]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 -- 172.21.15.37:0/1446425261 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 msgr2=0x7fafe4101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 --2- 172.21.15.37:0/1446425261 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4101c90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009910 tx=0x7fafd4009c20 comp rx=0 tx=0).stop 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 -- 172.21.15.37:0/1446425261 shutdown_connections 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 --2- 172.21.15.37:0/1446425261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 0x7fafe40fe300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 --2- 172.21.15.37:0/1446425261 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4101c90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 -- 172.21.15.37:0/1446425261 >> 172.21.15.37:0/1446425261 conn(0x7fafe40f9f10 msgr2=0x7fafe40fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 -- 172.21.15.37:0/1446425261 shutdown_connections 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.753+0000 7fafe88c4700 1 -- 172.21.15.37:0/1446425261 wait complete. 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe88c4700 1 Processor -- start 2024-04-15T22:53:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe88c4700 1 -- start start 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe88c4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe88c4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 0x7fafe40ff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe88c4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40ffb10 con 0x7fafe4101880 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe88c4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40ffc50 con 0x7fafe41021d0 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41832/0 (socket says 172.21.15.37:41832) 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe259c700 1 -- 172.21.15.37:0/74861710 learned_addr learned my addr 172.21.15.37:0/74861710 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe1d9b700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 0x7fafe40ff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe259c700 1 -- 172.21.15.37:0/74861710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 msgr2=0x7fafe40ff5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe259c700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 0x7fafe40ff5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe259c700 1 -- 172.21.15.37:0/74861710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd40095f0 con 0x7fafe4101880 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.754+0000 7fafe1d9b700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 0x7fafe40ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.755+0000 7fafe259c700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4100f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fafd4006010 tx=0x7fafd400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.755+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd401d070 con 0x7fafe4101880 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.755+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe40ffed0 con 0x7fafe4101880 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.755+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fafd400f460 con 0x7fafe4101880 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.755+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd4017630 con 0x7fafe4101880 2024-04-15T22:53:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.755+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe41003c0 con 0x7fafe4101880 2024-04-15T22:53:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.756+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7fafd4003a60 con 0x7fafe4101880 2024-04-15T22:53:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.756+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe4108d40 con 0x7fafe4101880 2024-04-15T22:53:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.757+0000 7fafd37fe700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fafcc06c600 0x7fafcc06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:53:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.757+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fafd408bb50 con 0x7fafe4101880 2024-04-15T22:53:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.757+0000 7fafe1d9b700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fafcc06c600 0x7fafcc06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:53:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.758+0000 7fafe1d9b700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fafcc06c600 0x7fafcc06eab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafd8005fd0 tx=0x7fafd8005ee0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:53:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.762+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fafd405ae20 con 0x7fafe4101880 2024-04-15T22:53:06.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:06.979+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fafe4105f70 con 0x7fafcc06c600 2024-04-15T22:53:07.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:07 smithi037 ceph-mon[29191]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:07.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:07 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:53:07.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:07 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:53:07.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:07 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:07.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:07 smithi189 ceph-mon[36179]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:07.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:07 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:53:07.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:07 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:53:07.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:07 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:08.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:08 smithi037 ceph-mon[29191]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:53:08.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:08 smithi189 ceph-mon[36179]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:53:09.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:09 smithi189 ceph-mon[36179]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:09.770 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:09 smithi037 ceph-mon[29191]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:10.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:10.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:10.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:10 smithi189 ceph-mon[36179]: Reconfiguring grafana.smithi037 (dependencies changed)... 2024-04-15T22:53:10.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:10 smithi189 ceph-mon[36179]: Reconfiguring daemon grafana.smithi037 on smithi037 2024-04-15T22:53:10.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:10 smithi189 ceph-mon[36179]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:11.031 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:11.031 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:11.031 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:10 smithi037 ceph-mon[29191]: Reconfiguring grafana.smithi037 (dependencies changed)... 2024-04-15T22:53:11.031 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:10 smithi037 ceph-mon[29191]: Reconfiguring daemon grafana.smithi037 on smithi037 2024-04-15T22:53:11.031 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:10 smithi037 ceph-mon[29191]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:11.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:11 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/75738744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1abff5-8aea-47ca-982a-29cd590fc31a"}]: dispatch 2024-04-15T22:53:11.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:11 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/75738744' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc1abff5-8aea-47ca-982a-29cd590fc31a"}]': finished 2024-04-15T22:53:11.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:11 smithi037 ceph-mon[29191]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:53:11.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:11 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:11.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:11 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2556564013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:53:11.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:11 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/75738744' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1abff5-8aea-47ca-982a-29cd590fc31a"}]: dispatch 2024-04-15T22:53:11.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:11 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/75738744' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc1abff5-8aea-47ca-982a-29cd590fc31a"}]': finished 2024-04-15T22:53:11.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:11 smithi189 ceph-mon[36179]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:53:11.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:11 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:11.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:11 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2556564013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:53:13.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:13 smithi037 ceph-mon[29191]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:13.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:13 smithi189 ceph-mon[36179]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:15.542 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:15 smithi037 ceph-mon[29191]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:15.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:15 smithi189 ceph-mon[36179]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:17.694 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:17 smithi037 ceph-mon[29191]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:17.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:17 smithi189 ceph-mon[36179]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:18.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:18 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:18.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:18 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:19.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:19 smithi037 ceph-mon[29191]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:19.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:19 smithi189 ceph-mon[36179]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:21.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:20 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:21.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:20 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:21.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:20 smithi189 ceph-mon[36179]: Reconfiguring prometheus.smithi037 (dependencies changed)... 2024-04-15T22:53:21.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:20 smithi189 ceph-mon[36179]: Reconfiguring daemon prometheus.smithi037 on smithi037 2024-04-15T22:53:21.218 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:20 smithi189 ceph-mon[36179]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:21.295 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:20 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:21.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:20 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:21.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:20 smithi037 ceph-mon[29191]: Reconfiguring prometheus.smithi037 (dependencies changed)... 2024-04-15T22:53:21.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:20 smithi037 ceph-mon[29191]: Reconfiguring daemon prometheus.smithi037 on smithi037 2024-04-15T22:53:21.296 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:20 smithi037 ceph-mon[29191]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:22.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:53:22.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:22.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:21 smithi037 ceph-mon[29191]: Deploying daemon osd.0 on smithi037 2024-04-15T22:53:22.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:53:22.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:22.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:21 smithi189 ceph-mon[36179]: Deploying daemon osd.0 on smithi037 2024-04-15T22:53:23.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:23 smithi037 ceph-mon[29191]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:23.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:23 smithi189 ceph-mon[36179]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:25.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:25 smithi189 ceph-mon[36179]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:25.669 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:25 smithi037 ceph-mon[29191]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:26.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:26.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:26.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: Reconfiguring ceph-exporter.smithi189 (monmap changed)... 2024-04-15T22:53:26.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-15T22:53:26.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:26.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: Reconfiguring daemon ceph-exporter.smithi189 on smithi189 2024-04-15T22:53:26.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:26 smithi037 ceph-mon[29191]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:26.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:26.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:26.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: Reconfiguring ceph-exporter.smithi189 (monmap changed)... 2024-04-15T22:53:26.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-15T22:53:26.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:26.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: Reconfiguring daemon ceph-exporter.smithi189 on smithi189 2024-04-15T22:53:26.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:26 smithi189 ceph-mon[36179]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:28.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:28 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:28.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:28 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:28.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:28 smithi037 ceph-mon[29191]: Reconfiguring crash.smithi189 (monmap changed)... 2024-04-15T22:53:28.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:28 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:53:28.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:28 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:28.291 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:28 smithi037 ceph-mon[29191]: Reconfiguring daemon crash.smithi189 on smithi189 2024-04-15T22:53:28.371 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:28 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:28.371 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:28 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:28.371 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:28 smithi189 ceph-mon[36179]: Reconfiguring crash.smithi189 (monmap changed)... 2024-04-15T22:53:28.371 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:28 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-15T22:53:28.371 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:28 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:28.371 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:28 smithi189 ceph-mon[36179]: Reconfiguring daemon crash.smithi189 on smithi189 2024-04-15T22:53:29.449 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:29 smithi189 ceph-mon[36179]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:29.449 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:29.449 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:29.449 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.lanqnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:53:29.449 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:53:29.449 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:29.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:29 smithi037 ceph-mon[29191]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:29.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:29.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:29.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.lanqnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:53:29.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:53:29.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: Reconfiguring mgr.smithi189.lanqnp (monmap changed)... 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: Reconfiguring daemon mgr.smithi189.lanqnp on smithi189 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:53:30.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:30.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: Reconfiguring mgr.smithi189.lanqnp (monmap changed)... 2024-04-15T22:53:30.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: Reconfiguring daemon mgr.smithi189.lanqnp on smithi189 2024-04-15T22:53:30.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:30.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:30.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:53:30.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:53:30.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:31.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:31 smithi037 ceph-mon[29191]: Reconfiguring mon.smithi189 (monmap changed)... 2024-04-15T22:53:31.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:31 smithi037 ceph-mon[29191]: Reconfiguring daemon mon.smithi189 on smithi189 2024-04-15T22:53:31.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:31 smithi037 ceph-mon[29191]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:31.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:31 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:31.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:31 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:31.582 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:31 smithi189 ceph-mon[36179]: Reconfiguring mon.smithi189 (monmap changed)... 2024-04-15T22:53:31.583 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:31 smithi189 ceph-mon[36179]: Reconfiguring daemon mon.smithi189 on smithi189 2024-04-15T22:53:31.583 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:31 smithi189 ceph-mon[36179]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:31.583 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:31 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:31.583 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:31 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:32.981 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:32.981 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi037.front.sepia.ceph.com:9093"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi037.front.sepia.ceph.com:9093"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi037.front.sepia.ceph.com:3000"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi037.front.sepia.ceph.com:3000"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:53:32.982 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:53:32.983 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi037.front.sepia.ceph.com:9095"}]: dispatch 2024-04-15T22:53:32.983 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi037.front.sepia.ceph.com:9095"}]: dispatch 2024-04-15T22:53:32.983 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:32.983 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:53:32.983 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:32 smithi037 ceph-mon[29191]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:33.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:33.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:33.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:53:33.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:53:33.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi037.front.sepia.ceph.com:9093"}]: dispatch 2024-04-15T22:53:33.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi037.front.sepia.ceph.com:9093"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi037.front.sepia.ceph.com:3000"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi037.front.sepia.ceph.com:3000"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi037.front.sepia.ceph.com:9095"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi037.front.sepia.ceph.com:9095"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:53:33.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:32 smithi189 ceph-mon[36179]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:33.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.902+0000 7fafd37fe700 1 -- 172.21.15.37:0/74861710 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fafe4105f70 con 0x7fafcc06c600 2024-04-15T22:53:33.897 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 0 on host 'smithi037' 2024-04-15T22:53:33.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fafcc06c600 msgr2=0x7fafcc06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:53:33.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fafcc06c600 0x7fafcc06eab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafd8005fd0 tx=0x7fafd8005ee0 comp rx=0 tx=0).stop 2024-04-15T22:53:33.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 msgr2=0x7fafe4100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:53:33.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4100f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fafd4006010 tx=0x7fafd400be20 comp rx=0 tx=0).stop 2024-04-15T22:53:33.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 shutdown_connections 2024-04-15T22:53:33.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fafcc06c600 0x7fafcc06eab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:33.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe4101880 0x7fafe4100f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:33.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 --2- 172.21.15.37:0/74861710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe41021d0 0x7fafe40ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:33.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.904+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 >> 172.21.15.37:0/74861710 conn(0x7fafe40f9f10 msgr2=0x7fafe4104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:53:33.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.905+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 shutdown_connections 2024-04-15T22:53:33.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:33.905+0000 7fafe88c4700 1 -- 172.21.15.37:0/74861710 wait complete. 2024-04-15T22:53:34.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:34.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.569 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.696 DEBUG:teuthology.orchestra.run.smithi037:osd.0> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@osd.0.service 2024-04-15T22:53:34.699 INFO:tasks.cephadm:Deploying osd.1 on smithi037 with /dev/vg_nvme/lv_3... 2024-04-15T22:53:34.699 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T22:53:34.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:34.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:34.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:35.647 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:35 smithi037 ceph-mon[29191]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:35.648 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:35 smithi037 ceph-mon[29191]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:53:35.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:35 smithi189 ceph-mon[36179]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:35.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:35 smithi189 ceph-mon[36179]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:53:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:53:36.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:36 smithi037 ceph-mon[29191]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:53:36.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:36 smithi037 ceph-mon[29191]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:53:36.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:36 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:36.550 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:36 smithi037 ceph-mon[29191]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-04-15T22:53:36.550 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:53:36 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-0[44091]: 2024-04-15T22:53:36.331+0000 7fcd84e8d700 -1 osd.0 0 waiting for initial osdmap 2024-04-15T22:53:36.550 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:53:36 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-0[44091]: 2024-04-15T22:53:36.338+0000 7fcd7bc78700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:53:36.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:36 smithi189 ceph-mon[36179]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:53:36.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:36 smithi189 ceph-mon[36179]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:53:36.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:36 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:36.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:36 smithi189 ceph-mon[36179]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-04-15T22:53:37.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:37 smithi189 ceph-mon[36179]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-04-15T22:53:37.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:37 smithi189 ceph-mon[36179]: osdmap e8: 1 total, 0 up, 1 in 2024-04-15T22:53:37.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:37.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:37.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:37 smithi189 ceph-mon[36179]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:37.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:37 smithi037 ceph-mon[29191]: from='osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-04-15T22:53:37.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:37 smithi037 ceph-mon[29191]: osdmap e8: 1 total, 0 up, 1 in 2024-04-15T22:53:37.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:37.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:37.819 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:37 smithi037 ceph-mon[29191]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:53:38.379 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:38 smithi037 ceph-mon[29191]: purged_snaps scrub starts 2024-04-15T22:53:38.380 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:38 smithi037 ceph-mon[29191]: purged_snaps scrub ok 2024-04-15T22:53:38.380 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:38 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:38.380 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:38 smithi037 ceph-mon[29191]: osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] boot 2024-04-15T22:53:38.380 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:38 smithi037 ceph-mon[29191]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:53:38.380 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:38 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:38.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:38 smithi189 ceph-mon[36179]: purged_snaps scrub starts 2024-04-15T22:53:38.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:38 smithi189 ceph-mon[36179]: purged_snaps scrub ok 2024-04-15T22:53:38.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:38 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:38.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:38 smithi189 ceph-mon[36179]: osd.0 [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] boot 2024-04-15T22:53:38.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:38 smithi189 ceph-mon[36179]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:53:38.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:38 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:53:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T22:53:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T22:53:38.804 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-04-15T22:53:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-04-15T22:53:38.805 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124978 s, 839 MB/s 2024-04-15T22:53:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-04-15T22:53:39.448 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:39 smithi037 ceph-mon[29191]: osdmap e10: 1 total, 1 up, 1 in 2024-04-15T22:53:39.448 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:39 smithi037 ceph-mon[29191]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:39.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi037:vg_nvme/lv_3 2024-04-15T22:53:39.667 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:53:39.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:39 smithi189 ceph-mon[36179]: osdmap e10: 1 total, 1 up, 1 in 2024-04-15T22:53:39.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:39 smithi189 ceph-mon[36179]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:40.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:40 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:40.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:40 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:40.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:40 smithi037 ceph-mon[29191]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:40.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:40 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:40.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:40 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:40.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:40 smithi189 ceph-mon[36179]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 -- 172.21.15.37:0/3241363377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480ff130 msgr2=0x7fa1480ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 --2- 172.21.15.37:0/3241363377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480ff130 0x7fa1480ff540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa138009910 tx=0x7fa138009c20 comp rx=0 tx=0).stop 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 -- 172.21.15.37:0/3241363377 shutdown_connections 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 --2- 172.21.15.37:0/3241363377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1481003d0 0x7fa148100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 --2- 172.21.15.37:0/3241363377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480ff130 0x7fa1480ff540 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 -- 172.21.15.37:0/3241363377 >> 172.21.15.37:0/3241363377 conn(0x7fa1480fa780 msgr2=0x7fa1480fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 -- 172.21.15.37:0/3241363377 shutdown_connections 2024-04-15T22:53:41.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.205+0000 7fa150094700 1 -- 172.21.15.37:0/3241363377 wait complete. 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa150094700 1 Processor -- start 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa150094700 1 -- start start 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa150094700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480ff130 0x7fa148103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa150094700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 0x7fa1481017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa150094700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148101cf0 con 0x7fa1481003d0 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa150094700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148101e30 con 0x7fa1480ff130 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14d62f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 0x7fa1481017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:53:41.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14d62f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 0x7fa1481017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35746/0 (socket says 172.21.15.37:35746) 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14d62f700 1 -- 172.21.15.37:0/1324280467 learned_addr learned my addr 172.21.15.37:0/1324280467 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14de30700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480ff130 0x7fa148103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14d62f700 1 -- 172.21.15.37:0/1324280467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480ff130 msgr2=0x7fa148103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14d62f700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480ff130 0x7fa148103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.206+0000 7fa14d62f700 1 -- 172.21.15.37:0/1324280467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1380095f0 con 0x7fa1481003d0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.207+0000 7fa14d62f700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 0x7fa1481017b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00b4b0 tx=0x7fa13c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.207+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa13c011800 con 0x7fa1481003d0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.207+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148102110 con 0x7fa1481003d0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.207+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa13c011e10 con 0x7fa1481003d0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.207+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa13c01a400 con 0x7fa1481003d0 2024-04-15T22:53:41.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.207+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14819d940 con 0x7fa1481003d0 2024-04-15T22:53:41.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.208+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa148107c90 con 0x7fa1481003d0 2024-04-15T22:53:41.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.208+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7fa13c010380 con 0x7fa1481003d0 2024-04-15T22:53:41.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.209+0000 7fa146ffd700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa13006c600 0x7fa13006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:53:41.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.209+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7fa13c08a5e0 con 0x7fa1481003d0 2024-04-15T22:53:41.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.209+0000 7fa14de30700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa13006c600 0x7fa13006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:53:41.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.209+0000 7fa14de30700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa13006c600 0x7fa13006eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa138006010 tx=0x7fa13801a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:53:41.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.214+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa13c0595d0 con 0x7fa1481003d0 2024-04-15T22:53:41.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:53:41.432+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa148068cd0 con 0x7fa13006c600 2024-04-15T22:53:41.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:41 smithi037 ceph-mon[29191]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:53:41.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:41 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:53:41.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:41 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:53:41.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:41 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:41.964 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:41 smithi189 ceph-mon[36179]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:53:41.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:41 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:53:41.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:41 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:53:41.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:41 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:43.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:43 smithi037 ceph-mon[29191]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:43.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:43 smithi189 ceph-mon[36179]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:45.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: Detected new or changed devices on smithi037 2024-04-15T22:53:45.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: Adjusting osd_memory_target on smithi037 to 11845M 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:53:45.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:45 smithi189 ceph-mon[36179]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:45.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: Detected new or changed devices on smithi037 2024-04-15T22:53:45.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:53:45.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: Adjusting osd_memory_target on smithi037 to 11845M 2024-04-15T22:53:45.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:45.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:53:45.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:45.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:53:45.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:45 smithi037 ceph-mon[29191]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:46.246 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:46 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3606944802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e875b0b4-8220-4f8a-9d7b-c2dffdd988dc"}]: dispatch 2024-04-15T22:53:46.246 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:46 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3606944802' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e875b0b4-8220-4f8a-9d7b-c2dffdd988dc"}]': finished 2024-04-15T22:53:46.246 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:46 smithi037 ceph-mon[29191]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:53:46.246 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:46 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:53:46.246 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:46 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2032786037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:53:46.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:46 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3606944802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e875b0b4-8220-4f8a-9d7b-c2dffdd988dc"}]: dispatch 2024-04-15T22:53:46.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:46 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3606944802' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e875b0b4-8220-4f8a-9d7b-c2dffdd988dc"}]': finished 2024-04-15T22:53:46.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:46 smithi189 ceph-mon[36179]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:53:46.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:46 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:53:46.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:46 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2032786037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:53:47.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:47 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:47.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:47 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:47.465 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:47 smithi189 ceph-mon[36179]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:47.494 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:47 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:47.494 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:47 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:47.494 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:47 smithi037 ceph-mon[29191]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:48.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:48 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:48.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:48 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:53:49.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:49 smithi037 ceph-mon[29191]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:49.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:49.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:49.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:49.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:53:49.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:49.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:49 smithi189 ceph-mon[36179]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:49.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:49.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:49.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:49.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:53:49.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:51.689 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:51 smithi037 ceph-mon[29191]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:51.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:51 smithi189 ceph-mon[36179]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:53.406 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:53 smithi037 ceph-mon[29191]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:53.406 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:53:53.406 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:53.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:53 smithi189 ceph-mon[36179]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:53.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:53:53.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:53:54.515 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:54 smithi037 ceph-mon[29191]: Deploying daemon osd.1 on smithi037 2024-04-15T22:53:54.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:54 smithi189 ceph-mon[36179]: Deploying daemon osd.1 on smithi037 2024-04-15T22:53:55.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:55 smithi037 ceph-mon[29191]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:55.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:55 smithi189 ceph-mon[36179]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:57.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:57 smithi189 ceph-mon[36179]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:57.724 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:57 smithi037 ceph-mon[29191]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:58.939 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:58.939 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:58.939 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:53:58.939 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:53:58 smithi037 ceph-mon[29191]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:53:59.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:59.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:53:59.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:53:59.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:53:58 smithi189 ceph-mon[36179]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:01.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:01.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:01.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:00 smithi037 ceph-mon[29191]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:01.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:01.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:01.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:00 smithi189 ceph-mon[36179]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:01.935 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:01 smithi037 ceph-mon[29191]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:54:02.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:01 smithi189 ceph-mon[36179]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:54:02.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.544+0000 7fa146ffd700 1 -- 172.21.15.37:0/1324280467 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa148068cd0 con 0x7fa13006c600 2024-04-15T22:54:02.539 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 1 on host 'smithi037' 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.547+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa13006c600 msgr2=0x7fa13006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.547+0000 7fa150094700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa13006c600 0x7fa13006eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa138006010 tx=0x7fa13801a040 comp rx=0 tx=0).stop 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 msgr2=0x7fa1481017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 0x7fa1481017b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00b4b0 tx=0x7fa13c00b870 comp rx=0 tx=0).stop 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 shutdown_connections 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa13006c600 0x7fa13006eab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:02.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1480ff130 0x7fa148103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:02.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 --2- 172.21.15.37:0/1324280467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1481003d0 0x7fa1481017b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:02.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 >> 172.21.15.37:0/1324280467 conn(0x7fa1480fa780 msgr2=0x7fa1480685c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:54:02.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 shutdown_connections 2024-04-15T22:54:02.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:02.548+0000 7fa150094700 1 -- 172.21.15.37:0/1324280467 wait complete. 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: osdmap e13: 2 total, 1 up, 2 in 2024-04-15T22:54:03.109 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:02 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:03.135 DEBUG:teuthology.orchestra.run.smithi037:osd.1> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@osd.1.service 2024-04-15T22:54:03.138 INFO:tasks.cephadm:Deploying osd.2 on smithi037 with /dev/vg_nvme/lv_2... 2024-04-15T22:54:03.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:03.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:03.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-04-15T22:54:03.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: osdmap e13: 2 total, 1 up, 2 in 2024-04-15T22:54:03.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:02 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:04.179 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:04.179 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:04.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:04.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:54:04.911 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: purged_snaps scrub starts 2024-04-15T22:54:04.911 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: purged_snaps scrub ok 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] boot 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:04.912 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:05.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: purged_snaps scrub starts 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: purged_snaps scrub ok 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: osd.1 [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] boot 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:05.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:06.683 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T22:54:06.683 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T22:54:06.683 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10+0 records in 2024-04-15T22:54:06.683 INFO:teuthology.orchestra.run.smithi037.stderr:10+0 records out 2024-04-15T22:54:06.683 INFO:teuthology.orchestra.run.smithi037.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112666 s, 931 MB/s 2024-04-15T22:54:06.683 INFO:teuthology.orchestra.run.smithi037.stderr:--> Zapping successful for: 2024-04-15T22:54:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:06 smithi037 ceph-mon[29191]: osdmap e15: 2 total, 2 up, 2 in 2024-04-15T22:54:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:06 smithi037 ceph-mon[29191]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:06.918 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:06 smithi189 ceph-mon[36179]: osdmap e15: 2 total, 2 up, 2 in 2024-04-15T22:54:06.918 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:06 smithi189 ceph-mon[36179]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:07.297 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi037:vg_nvme/lv_2 2024-04-15T22:54:08.160 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:54:09.689 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:09 smithi037 ceph-mon[29191]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:09.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:09 smithi189 ceph-mon[36179]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/607674023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 msgr2=0x7f6fec105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/607674023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec105a80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009910 tx=0x7f6fd8009c20 comp rx=0 tx=0).stop 2024-04-15T22:54:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/607674023 shutdown_connections 2024-04-15T22:54:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/607674023 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec105a80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/607674023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fec100d80 0x7f6fec103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/607674023 >> 172.21.15.37:0/607674023 conn(0x7f6fec0fa7a0 msgr2=0x7f6fec0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/607674023 shutdown_connections 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.205+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/607674023 wait complete. 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff2d2b700 1 Processor -- start 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff2d2b700 1 -- start start 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff2d2b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fec100d80 0x7f6fec1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff2d2b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec0fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff2d2b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0ff270 con 0x7f6fec1036a0 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff2d2b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0ff3b0 con 0x7f6fec100d80 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6febfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec0fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6febfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec0fed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59150/0 (socket says 172.21.15.37:59150) 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6febfff700 1 -- 172.21.15.37:0/3541693987 learned_addr learned my addr 172.21.15.37:0/3541693987 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6ff0ac7700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fec100d80 0x7f6fec1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:54:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6febfff700 1 -- 172.21.15.37:0/3541693987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fec100d80 msgr2=0x7f6fec1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6febfff700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fec100d80 0x7f6fec1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.206+0000 7f6febfff700 1 -- 172.21.15.37:0/3541693987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80095f0 con 0x7f6fec1036a0 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.207+0000 7f6febfff700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec0fed30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8000c00 tx=0x7f6fd800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.207+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd801d070 con 0x7f6fec1036a0 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.207+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6fd8005dc0 con 0x7f6fec1036a0 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.207+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec0ff630 con 0x7f6fec1036a0 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.207+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd80226a0 con 0x7f6fec1036a0 2024-04-15T22:54:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.207+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec0ffb20 con 0x7f6fec1036a0 2024-04-15T22:54:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.208+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec18fc00 con 0x7f6fec1036a0 2024-04-15T22:54:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.208+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f6fd80038f0 con 0x7f6fec1036a0 2024-04-15T22:54:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.209+0000 7f6fe9ffb700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6fdc06c600 0x7f6fdc06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:54:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.209+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f6fd808c0b0 con 0x7f6fec1036a0 2024-04-15T22:54:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.209+0000 7f6ff0ac7700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6fdc06c600 0x7f6fdc06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:54:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.209+0000 7f6ff0ac7700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6fdc06c600 0x7f6fdc06eab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6fec069890 tx=0x7f6fe0009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:54:10.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.214+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6fd805ae50 con 0x7f6fec1036a0 2024-04-15T22:54:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:10.433+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fec02cf60 con 0x7f6fdc06c600 2024-04-15T22:54:10.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: Detected new or changed devices on smithi037 2024-04-15T22:54:10.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: Adjusting osd_memory_target on smithi037 to 5922M 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:54:10.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:10 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: Detected new or changed devices on smithi037 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: Adjusting osd_memory_target on smithi037 to 5922M 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:10.808 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi037:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:54:10.809 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:54:10.809 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:54:10.809 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:10 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:12.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:12.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:12.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:12 smithi037 ceph-mon[29191]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:12.964 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:12.964 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:12.964 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:12 smithi189 ceph-mon[36179]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:14.605 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:14 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/605429412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08257888-e768-4845-9efa-8358f6e8caec"}]: dispatch 2024-04-15T22:54:14.605 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:14 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/605429412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08257888-e768-4845-9efa-8358f6e8caec"}]': finished 2024-04-15T22:54:14.605 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:14 smithi037 ceph-mon[29191]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:54:14.605 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:14 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:14.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:14 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/605429412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08257888-e768-4845-9efa-8358f6e8caec"}]: dispatch 2024-04-15T22:54:14.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:14 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/605429412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08257888-e768-4845-9efa-8358f6e8caec"}]': finished 2024-04-15T22:54:14.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:14 smithi189 ceph-mon[36179]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:54:14.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:14 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:15.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:15 smithi037 ceph-mon[29191]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:15.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:15 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1775482345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:54:15.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:15 smithi189 ceph-mon[36179]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:15.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:15 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1775482345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:54:17.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:16 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:17.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:16 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:17.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:16 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:17.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:16 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:17.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:16 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:17.215 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:16 smithi189 ceph-mon[36179]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:17.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:16 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:17.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:16 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:17.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:16 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:17.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:16 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:17.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:16 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:17.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:16 smithi037 ceph-mon[29191]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:18.549 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:18 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:18.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:18 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:19.705 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:19 smithi037 ceph-mon[29191]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:19.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:19 smithi189 ceph-mon[36179]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:21.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:21 smithi189 ceph-mon[36179]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:21.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:54:21.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:21.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:21 smithi037 ceph-mon[29191]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:21.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:54:21.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:22.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:22 smithi189 ceph-mon[36179]: Deploying daemon osd.2 on smithi037 2024-04-15T22:54:22.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:22 smithi037 ceph-mon[29191]: Deploying daemon osd.2 on smithi037 2024-04-15T22:54:23.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:23 smithi189 ceph-mon[36179]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:23.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:23 smithi037 ceph-mon[29191]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:25.483 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:25 smithi189 ceph-mon[36179]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:25.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:25 smithi037 ceph-mon[29191]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:27.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:27 smithi189 ceph-mon[36179]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:27.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:27 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:27.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:27 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:27.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:27 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:27.789 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:27 smithi037 ceph-mon[29191]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:27.789 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:27 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:27.789 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:27 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:27.789 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:27 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:29.568 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:29 smithi037 ceph-mon[29191]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:29.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:29 smithi189 ceph-mon[36179]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:30.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.617+0000 7f6fe9ffb700 1 -- 172.21.15.37:0/3541693987 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6fec02cf60 con 0x7f6fdc06c600 2024-04-15T22:54:30.612 INFO:teuthology.orchestra.run.smithi037.stdout:Created osd(s) 2 on host 'smithi037' 2024-04-15T22:54:30.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.619+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6fdc06c600 msgr2=0x7f6fdc06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.619+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6fdc06c600 0x7f6fdc06eab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6fec069890 tx=0x7f6fe0009260 comp rx=0 tx=0).stop 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.619+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 msgr2=0x7f6fec0fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.619+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec0fed30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8000c00 tx=0x7f6fd800f690 comp rx=0 tx=0).stop 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 shutdown_connections 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6fdc06c600 0x7f6fdc06eab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fec100d80 0x7f6fec1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 --2- 172.21.15.37:0/3541693987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fec1036a0 0x7f6fec0fed30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 >> 172.21.15.37:0/3541693987 conn(0x7f6fec0fa7a0 msgr2=0x7f6fec0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 shutdown_connections 2024-04-15T22:54:30.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:54:30.620+0000 7f6ff2d2b700 1 -- 172.21.15.37:0/3541693987 wait complete. 2024-04-15T22:54:31.494 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:31 smithi037 ceph-mon[29191]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:31.495 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:31 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:31.495 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:31 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:31.495 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:31 smithi037 ceph-mon[29191]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:54:31.521 DEBUG:teuthology.orchestra.run.smithi037:osd.2> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@osd.2.service 2024-04-15T22:54:31.523 INFO:tasks.cephadm:Deploying osd.3 on smithi189 with /dev/vg_nvme/lv_4... 2024-04-15T22:54:31.524 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T22:54:31.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:31 smithi189 ceph-mon[36179]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:31.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:31 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:31.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:31 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:31.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:31 smithi189 ceph-mon[36179]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:54:31.742 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:54:32.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-mon[29191]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:54:32.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-mon[29191]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:54:32.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:32.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-mon[29191]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-04-15T22:54:32.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-mon[29191]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:32.818 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-2[55172]: 2024-04-15T22:54:32.593+0000 7fc5a5cda700 -1 osd.2 0 waiting for initial osdmap 2024-04-15T22:54:32.818 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:32 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-2[55172]: 2024-04-15T22:54:32.598+0000 7fc59f2ca700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:54:32.866 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:32 smithi189 ceph-mon[36179]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:54:32.866 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:32 smithi189 ceph-mon[36179]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:54:32.866 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:32.866 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:32 smithi189 ceph-mon[36179]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]: dispatch 2024-04-15T22:54:32.866 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:32 smithi189 ceph-mon[36179]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T22:54:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T22:54:33.687 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-15T22:54:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-15T22:54:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116128 s, 903 MB/s 2024-04-15T22:54:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-15T22:54:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:33 smithi189 ceph-mon[36179]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-04-15T22:54:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:33 smithi189 ceph-mon[36179]: osdmap e18: 3 total, 2 up, 3 in 2024-04-15T22:54:33.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:33 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:33.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:33 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:33.966 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:33 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:34.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:33 smithi037 ceph-mon[29191]: from='osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi037", "root=default"]}]': finished 2024-04-15T22:54:34.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:33 smithi037 ceph-mon[29191]: osdmap e18: 3 total, 2 up, 3 in 2024-04-15T22:54:34.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:33 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:34.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:33 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:34.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:33 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:34.362 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-04-15T22:54:34.568 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: purged_snaps scrub starts 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: purged_snaps scrub ok 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] boot 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:34.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:35.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: purged_snaps scrub starts 2024-04-15T22:54:35.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: purged_snaps scrub ok 2024-04-15T22:54:35.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: osd.2 [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] boot 2024-04-15T22:54:35.067 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:54:35.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:54:35.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:35.068 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:35.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:35.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:54:35.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-15T22:54:35.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:54:35.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:54:35.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:54:35.818 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:35 smithi037 ceph-mon[29191]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:54:35.901 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:54:35.901 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:54:35.902 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-15T22:54:35.902 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:54:35.902 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:54:35.902 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:54:35.902 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:35 smithi189 ceph-mon[36179]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:54:36.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 -- 172.21.15.189:0/1534416583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 msgr2=0x7fe504100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:36.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 --2- 172.21.15.189:0/1534416583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504100650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009910 tx=0x7fe4f8009c20 comp rx=0 tx=0).stop 2024-04-15T22:54:36.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 -- 172.21.15.189:0/1534416583 shutdown_connections 2024-04-15T22:54:36.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 --2- 172.21.15.189:0/1534416583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe504101440 0x7fe504101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:36.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 --2- 172.21.15.189:0/1534416583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504100650 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:36.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 -- 172.21.15.189:0/1534416583 >> 172.21.15.189:0/1534416583 conn(0x7fe5040fb7f0 msgr2=0x7fe5040fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:54:36.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 -- 172.21.15.189:0/1534416583 shutdown_connections 2024-04-15T22:54:36.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 -- 172.21.15.189:0/1534416583 wait complete. 2024-04-15T22:54:36.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 Processor -- start 2024-04-15T22:54:36.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 -- start start 2024-04-15T22:54:36.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.126+0000 7fe50af0c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:54:36.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe50af0c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe504101440 0x7fe504196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:54:36.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe50af0c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504196640 con 0x7fe504101440 2024-04-15T22:54:36.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe50af0c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504196780 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46106/0 (socket says 172.21.15.189:46106) 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 -- 172.21.15.189:0/3514696487 learned_addr learned my addr 172.21.15.189:0/3514696487 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe503fff700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe504101440 0x7fe504196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 -- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe504101440 msgr2=0x7fe504196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe504101440 0x7fe504196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 -- 172.21.15.189:0/3514696487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f80095f0 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe508ca8700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504195ae0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8005b40 tx=0x7fe4f800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f801d070 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe4f8005dc0 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50419b1d0 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.127+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f8022470 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.128+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50419b6c0 con 0x7fe504100240 2024-04-15T22:54:36.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.128+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe504108d90 con 0x7fe504100240 2024-04-15T22:54:36.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.129+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7fe4f8003680 con 0x7fe504100240 2024-04-15T22:54:36.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.129+0000 7fe501ffb700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4f406c610 0x7fe4f406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:54:36.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.130+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3232+0+0 (secure 0 0 0) 0x7fe4f808c350 con 0x7fe504100240 2024-04-15T22:54:36.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.130+0000 7fe503fff700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4f406c610 0x7fe4f406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:54:36.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.130+0000 7fe503fff700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4f406c610 0x7fe4f406eac0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009c60 tx=0x7fe4f0009260 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:54:36.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.134+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe4f805aee0 con 0x7fe504100240 2024-04-15T22:54:36.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:36.334+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe50419bd50 con 0x7fe4f406c610 2024-04-15T22:54:36.928 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:36 smithi189 ceph-mon[36179]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:54:36.928 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:36 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:54:36.928 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:36 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:54:36.928 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:36 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:36.929 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:36 smithi189 ceph-mon[36179]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:37.022 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:36 smithi037 sudo[57493]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:54:37.023 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:36 smithi037 sudo[57493]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:37.023 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:36 smithi037 sudo[57493]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:37.023 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:36 smithi037 sudo[57493]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:37.023 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:36 smithi037 ceph-mon[29191]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:54:37.023 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:36 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:54:37.024 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:36 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:54:37.024 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:36 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:37.024 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:36 smithi037 ceph-mon[29191]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:37.310 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57565]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:54:37.310 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:37.311 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57565]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:37.311 INFO:journalctl@ceph.osd.0.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57565]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:37.567 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57648]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:54:37.567 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57648]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:37.567 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57648]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:37.568 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57648]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:37.841 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57671]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:54:37.841 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57671]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:37.841 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57671]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:37.841 INFO:journalctl@ceph.osd.1.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57671]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: Detected new or changed devices on smithi037 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: Adjusting osd_memory_target on smithi037 to 3948M 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.842 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:37.843 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.872 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:54:37.872 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: Detected new or changed devices on smithi037 2024-04-15T22:54:37.872 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.872 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.872 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: Adjusting osd_memory_target on smithi037 to 3948M 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:37.873 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:38.118 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57681]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:54:38.119 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57681]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:38.119 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57681]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:38.119 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:37 smithi037 sudo[57681]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:38.379 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57754]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:54:38.379 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57754]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:38.379 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57754]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:38.379 INFO:journalctl@ceph.osd.2.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57754]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:38.646 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57810]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:54:38.646 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:38.646 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57810]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:38.646 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57810]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:39.062 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57814]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:54:39.063 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57814]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:39.063 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57814]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:39.063 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:38 smithi037 sudo[57814]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:39.148 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:38 smithi189 sudo[40125]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:54:39.148 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:38 smithi189 sudo[40125]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:39.149 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:38 smithi189 sudo[40125]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:39.149 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:38 smithi189 sudo[40125]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:39.174+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe4f8057820 con 0x7fe504100240 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:54:39.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:39 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 sudo[40128]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 sudo[40128]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 sudo[40128]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 sudo[40128]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-04-15T22:54:39.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:39 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-15T22:54:40.469 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:54:40.469 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:54:40.469 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: mgrmap e19: smithi037.gtwkck(active, since 2m), standbys: smithi189.lanqnp 2024-04-15T22:54:40.469 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:40.469 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/3883299024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d49e4a2-0131-436b-b02c-77dd4237bee8"}]: dispatch 2024-04-15T22:54:40.469 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d49e4a2-0131-436b-b02c-77dd4237bee8"}]: dispatch 2024-04-15T22:54:40.470 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d49e4a2-0131-436b-b02c-77dd4237bee8"}]': finished 2024-04-15T22:54:40.470 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:54:40.470 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:40.470 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:40 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/4156464427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:54:40.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:54:40.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:54:40.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: mgrmap e19: smithi037.gtwkck(active, since 2m), standbys: smithi189.lanqnp 2024-04-15T22:54:40.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:40.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/3883299024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d49e4a2-0131-436b-b02c-77dd4237bee8"}]: dispatch 2024-04-15T22:54:40.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d49e4a2-0131-436b-b02c-77dd4237bee8"}]: dispatch 2024-04-15T22:54:40.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d49e4a2-0131-436b-b02c-77dd4237bee8"}]': finished 2024-04-15T22:54:40.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:54:40.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:40.715 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:40 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/4156464427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:54:41.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:41 smithi189 ceph-mon[36179]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:41.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:41 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:41.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:41 smithi037 ceph-mon[29191]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:41.562 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:41 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:42.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:42 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:42.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:42 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:42.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:42 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:42.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:42 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:42.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:42.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:42.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:42.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:42 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:43.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:43 smithi037 ceph-mon[29191]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:43.647 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:43 smithi189 ceph-mon[36179]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:45.663 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:45 smithi189 ceph-mon[36179]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:45.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:45 smithi037 ceph-mon[29191]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:47.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:47 smithi189 ceph-mon[36179]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:47.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:47 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:54:47.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:47 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:47.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:47 smithi037 ceph-mon[29191]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:47.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:47 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:54:47.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:47 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:48.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:48 smithi189 ceph-mon[36179]: Deploying daemon osd.3 on smithi189 2024-04-15T22:54:48.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:48 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:48.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:48 smithi037 ceph-mon[29191]: Deploying daemon osd.3 on smithi189 2024-04-15T22:54:48.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:48 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:54:49.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:49 smithi189 ceph-mon[36179]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:49.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:49 smithi037 ceph-mon[29191]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:51.693 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:51 smithi189 ceph-mon[36179]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:51.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:51 smithi037 ceph-mon[29191]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:53.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:53.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:53.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:53.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:52 smithi189 ceph-mon[36179]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:53.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:53.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:53.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:53.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:52 smithi037 ceph-mon[29191]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:55.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:55 smithi189 ceph-mon[36179]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:55.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:55.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:55.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:55.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:55.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:55 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:55.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:55 smithi037 ceph-mon[29191]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:55.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:55.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:55.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:54:55.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:54:55.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:55 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:56.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.098+0000 7fe501ffb700 1 -- 172.21.15.189:0/3514696487 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe50419bd50 con 0x7fe4f406c610 2024-04-15T22:54:56.092 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 3 on host 'smithi189' 2024-04-15T22:54:56.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4f406c610 msgr2=0x7fe4f406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:56.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4f406c610 0x7fe4f406eac0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009c60 tx=0x7fe4f0009260 comp rx=0 tx=0).stop 2024-04-15T22:54:56.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 msgr2=0x7fe504195ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:54:56.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504195ae0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8005b40 tx=0x7fe4f800fa00 comp rx=0 tx=0).stop 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 shutdown_connections 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4f406c610 0x7fe4f406eac0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe504100240 0x7fe504195ae0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 --2- 172.21.15.189:0/3514696487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe504101440 0x7fe504196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 >> 172.21.15.189:0/3514696487 conn(0x7fe5040fb7f0 msgr2=0x7fe504104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 shutdown_connections 2024-04-15T22:54:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:54:56.102+0000 7fe50af0c700 1 -- 172.21.15.189:0/3514696487 wait complete. 2024-04-15T22:54:56.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:56 smithi189 ceph-mon[36179]: from='osd.3 [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:54:56.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:56 smithi189 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:54:56.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:56 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:56.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:56 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:56.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:56 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:56.738 DEBUG:teuthology.orchestra.run.smithi189:osd.3> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@osd.3.service 2024-04-15T22:54:56.740 INFO:tasks.cephadm:Deploying osd.4 on smithi189 with /dev/vg_nvme/lv_3... 2024-04-15T22:54:56.740 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T22:54:56.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:56 smithi037 ceph-mon[29191]: from='osd.3 [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:54:56.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:56 smithi037 ceph-mon[29191]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:54:56.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:56 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:56.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:56 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:54:56.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:56 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:54:57.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-mon[36179]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:57.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:54:57.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-mon[36179]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:54:57.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:57.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-mon[36179]: from='osd.3 [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:54:57.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:54:57.714 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:54:57.434+0000 7f9f46757700 -1 osd.3 0 waiting for initial osdmap 2024-04-15T22:54:57.715 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:54:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:54:57.440+0000 7f9f4154a700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:54:57.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:57 smithi037 ceph-mon[29191]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:57.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:57 smithi037 ceph-mon[29191]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:54:57.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:57 smithi037 ceph-mon[29191]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:54:57.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:57 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:57.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:57 smithi037 ceph-mon[29191]: from='osd.3 [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:54:57.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:57 smithi037 ceph-mon[29191]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:54:57.932 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:54:58.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:58 smithi189 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-15T22:54:58.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:58 smithi189 ceph-mon[36179]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:54:58.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:58.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:58 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:58.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:58 smithi037 ceph-mon[29191]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-15T22:54:58.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:58 smithi037 ceph-mon[29191]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:54:58.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:58.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:58 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: purged_snaps scrub starts 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: purged_snaps scrub ok 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: osd.3 [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] boot 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:54:59.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:54:59 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: purged_snaps scrub starts 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: purged_snaps scrub ok 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: osd.3 [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] boot 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:54:59.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:54:59 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:55:00.535 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:00.536 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:55:00.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:00.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:00.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:00.719 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T22:55:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T22:55:00.850 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-15T22:55:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-15T22:55:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127093 s, 825 MB/s 2024-04-15T22:55:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-15T22:55:01.453 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-04-15T22:55:01.659 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:55:01.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:01 smithi189 ceph-mon[36179]: Detected new or changed devices on smithi189 2024-04-15T22:55:01.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:01 smithi189 ceph-mon[36179]: Adjusting osd_memory_target on smithi189 to 14912M 2024-04-15T22:55:01.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:01 smithi189 ceph-mon[36179]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:01.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:01 smithi189 ceph-mon[36179]: osdmap e28: 4 total, 4 up, 4 in 2024-04-15T22:55:01.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:01 smithi037 ceph-mon[29191]: Detected new or changed devices on smithi189 2024-04-15T22:55:01.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:01 smithi037 ceph-mon[29191]: Adjusting osd_memory_target on smithi189 to 14912M 2024-04-15T22:55:01.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:01 smithi037 ceph-mon[29191]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:01.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:01 smithi037 ceph-mon[29191]: osdmap e28: 4 total, 4 up, 4 in 2024-04-15T22:55:02.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:02 smithi189 ceph-mon[36179]: osdmap e29: 4 total, 4 up, 4 in 2024-04-15T22:55:02.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:02 smithi037 ceph-mon[29191]: osdmap e29: 4 total, 4 up, 4 in 2024-04-15T22:55:03.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 -- 172.21.15.189:0/1941121388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 msgr2=0x7f3a74103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:03.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/1941121388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74103140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009910 tx=0x7f3a64009c20 comp rx=0 tx=0).stop 2024-04-15T22:55:03.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 -- 172.21.15.189:0/1941121388 shutdown_connections 2024-04-15T22:55:03.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/1941121388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a74103680 0x7f3a74105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:03.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/1941121388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74103140 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 -- 172.21.15.189:0/1941121388 >> 172.21.15.189:0/1941121388 conn(0x7f3a740fa780 msgr2=0x7f3a740fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 -- 172.21.15.189:0/1941121388 shutdown_connections 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 -- 172.21.15.189:0/1941121388 wait complete. 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 Processor -- start 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.275+0000 7f3a79ad6700 1 -- start start 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a79ad6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a79ad6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a74103680 0x7f3a74195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a79ad6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a741965c0 con 0x7f3a74103680 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a79ad6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a74196700 con 0x7f3a74100d60 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40814/0 (socket says 172.21.15.189:40814) 2024-04-15T22:55:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 -- 172.21.15.189:0/522604931 learned_addr learned my addr 172.21.15.189:0/522604931 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a72ffd700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a74103680 0x7f3a74195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 -- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a74103680 msgr2=0x7f3a74195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a74103680 0x7f3a74195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 -- 172.21.15.189:0/522604931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640095f0 con 0x7f3a74100d60 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a737fe700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74195a60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400bda0 tx=0x7f3a6400be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a6401d070 con 0x7f3a74100d60 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3a6400f460 con 0x7f3a74100d60 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a64021640 con 0x7f3a74100d60 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.276+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7419b150 con 0x7f3a74100d60 2024-04-15T22:55:03.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.277+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7419b640 con 0x7f3a74100d60 2024-04-15T22:55:03.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.277+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7418fbc0 con 0x7f3a74100d60 2024-04-15T22:55:03.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.278+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3a6400fa00 con 0x7f3a74100d60 2024-04-15T22:55:03.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.278+0000 7f3a70ff9700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3a5c06c680 0x7f3a5c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:03.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.278+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f3a6408c400 con 0x7f3a74100d60 2024-04-15T22:55:03.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.279+0000 7f3a72ffd700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3a5c06c680 0x7f3a5c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:03.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.279+0000 7f3a72ffd700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3a5c06c680 0x7f3a5c06eb30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3a68005950 tx=0x7f3a680058e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:03.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.283+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3a6405ac10 con 0x7f3a74100d60 2024-04-15T22:55:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:03.482+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a74061190 con 0x7f3a5c06c680 2024-04-15T22:55:03.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:03 smithi189 ceph-mon[36179]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:03.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:03 smithi189 ceph-mon[36179]: osdmap e30: 4 total, 4 up, 4 in 2024-04-15T22:55:03.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:03.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:03 smithi037 ceph-mon[29191]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:03.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:03 smithi037 ceph-mon[29191]: osdmap e30: 4 total, 4 up, 4 in 2024-04-15T22:55:03.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:04.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:04 smithi189 ceph-mon[36179]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:55:04.632 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:55:04.632 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:55:04.632 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:04.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:04 smithi037 ceph-mon[29191]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:55:04.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:55:04.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:55:04.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:05.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:05 smithi037 ceph-mon[29191]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:05.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:05 smithi189 ceph-mon[36179]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:06.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:06 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/1612271653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b93aee26-d0dd-4cbc-b45e-b005033c1a26"}]: dispatch 2024-04-15T22:55:06.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:06 smithi189 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b93aee26-d0dd-4cbc-b45e-b005033c1a26"}]: dispatch 2024-04-15T22:55:06.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:06 smithi189 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b93aee26-d0dd-4cbc-b45e-b005033c1a26"}]': finished 2024-04-15T22:55:06.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:06 smithi189 ceph-mon[36179]: osdmap e31: 5 total, 4 up, 5 in 2024-04-15T22:55:06.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:06 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:06 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/1612271653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b93aee26-d0dd-4cbc-b45e-b005033c1a26"}]: dispatch 2024-04-15T22:55:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:06 smithi037 ceph-mon[29191]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b93aee26-d0dd-4cbc-b45e-b005033c1a26"}]: dispatch 2024-04-15T22:55:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:06 smithi037 ceph-mon[29191]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b93aee26-d0dd-4cbc-b45e-b005033c1a26"}]': finished 2024-04-15T22:55:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:06 smithi037 ceph-mon[29191]: osdmap e31: 5 total, 4 up, 5 in 2024-04-15T22:55:06.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:06 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:07.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:07 smithi037 ceph-mon[29191]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-15T22:55:07.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:07 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2908839408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:55:07.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:07 smithi189 ceph-mon[36179]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-15T22:55:07.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:07 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/2908839408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:55:09.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:09 smithi037 ceph-mon[29191]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T22:55:09.869 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:09 smithi189 ceph-mon[36179]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T22:55:11.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:11 smithi037 ceph-mon[29191]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-15T22:55:11.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:11 smithi189 ceph-mon[36179]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-15T22:55:13.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:13 smithi037 ceph-mon[29191]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-15T22:55:13.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:13 smithi189 ceph-mon[36179]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-15T22:55:14.463 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:14 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:55:14.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:14 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:14.464 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:14 smithi189 ceph-mon[36179]: Deploying daemon osd.4 on smithi189 2024-04-15T22:55:14.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:14 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:55:14.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:14 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:14.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:14 smithi037 ceph-mon[29191]: Deploying daemon osd.4 on smithi189 2024-04-15T22:55:15.437 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:15 smithi189 ceph-mon[36179]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-15T22:55:15.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:15 smithi037 ceph-mon[29191]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-15T22:55:17.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:17 smithi189 ceph-mon[36179]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:17.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:17 smithi037 ceph-mon[29191]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:18.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:18 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:18.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:18 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:19.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:19 smithi189 ceph-mon[36179]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:19.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:19.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:19.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:19.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:19 smithi037 ceph-mon[29191]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:19.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:19.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:19.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:21.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:21 smithi189 ceph-mon[36179]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:21.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:21 smithi037 ceph-mon[29191]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.096+0000 7f3a70ff9700 1 -- 172.21.15.189:0/522604931 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3a74061190 con 0x7f3a5c06c680 2024-04-15T22:55:22.091 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 4 on host 'smithi189' 2024-04-15T22:55:22.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3a5c06c680 msgr2=0x7f3a5c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:22.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3a5c06c680 0x7f3a5c06eb30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3a68005950 tx=0x7f3a680058e0 comp rx=0 tx=0).stop 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 msgr2=0x7f3a74195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74195a60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400bda0 tx=0x7f3a6400be80 comp rx=0 tx=0).stop 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 shutdown_connections 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3a5c06c680 0x7f3a5c06eb30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a74100d60 0x7f3a74195a60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 --2- 172.21.15.189:0/522604931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a74103680 0x7f3a74195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 >> 172.21.15.189:0/522604931 conn(0x7f3a740fa780 msgr2=0x7f3a740fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 shutdown_connections 2024-04-15T22:55:22.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:22.098+0000 7f3a79ad6700 1 -- 172.21.15.189:0/522604931 wait complete. 2024-04-15T22:55:23.021 DEBUG:teuthology.orchestra.run.smithi189:osd.4> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@osd.4.service 2024-04-15T22:55:23.024 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_2... 2024-04-15T22:55:23.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T22:55:23.279 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:55:23.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:23.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:23.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:23 smithi037 ceph-mon[29191]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:23.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:23 smithi037 ceph-mon[29191]: from='osd.4 [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:55:23.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:23 smithi037 ceph-mon[29191]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:55:23.402 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:23 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:23.402 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:23 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:23.402 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:23 smithi189 ceph-mon[36179]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:23.402 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:23 smithi189 ceph-mon[36179]: from='osd.4 [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:55:23.402 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:23 smithi189 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: osdmap e32: 5 total, 4 up, 5 in 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: from='osd.4 [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:24.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: osdmap e32: 5 total, 4 up, 5 in 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: from='osd.4 [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:24.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:24 smithi037 ceph-mon[29191]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:24.649 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:55:24.354+0000 7f83b8155700 -1 osd.4 0 waiting for initial osdmap 2024-04-15T22:55:24.649 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:55:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:55:24.362+0000 7f83b3749700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:55:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T22:55:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T22:55:25.155 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-15T22:55:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-15T22:55:25.156 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140112 s, 748 MB/s 2024-04-15T22:55:25.156 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-15T22:55:25.562 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:25 smithi189 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-15T22:55:25.563 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:25 smithi189 ceph-mon[36179]: osdmap e33: 5 total, 4 up, 5 in 2024-04-15T22:55:25.563 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:25 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:25.563 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:25 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:25.563 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:25 smithi189 ceph-mon[36179]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:25.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:25 smithi037 ceph-mon[29191]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-15T22:55:25.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:25 smithi037 ceph-mon[29191]: osdmap e33: 5 total, 4 up, 5 in 2024-04-15T22:55:25.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:25.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:25.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:25 smithi037 ceph-mon[29191]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:55:26.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:26 smithi189 ceph-mon[36179]: purged_snaps scrub starts 2024-04-15T22:55:26.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:26 smithi189 ceph-mon[36179]: purged_snaps scrub ok 2024-04-15T22:55:26.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:26 smithi189 ceph-mon[36179]: osd.4 [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] boot 2024-04-15T22:55:26.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:26 smithi189 ceph-mon[36179]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:55:26.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:26 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:26.760 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-04-15T22:55:26.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:26 smithi037 ceph-mon[29191]: purged_snaps scrub starts 2024-04-15T22:55:26.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:26 smithi037 ceph-mon[29191]: purged_snaps scrub ok 2024-04-15T22:55:26.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:26 smithi037 ceph-mon[29191]: osd.4 [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] boot 2024-04-15T22:55:26.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:26 smithi037 ceph-mon[29191]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:55:26.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:26 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:55:27.160 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:55:27.625 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:27 smithi189 ceph-mon[36179]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:55:27.625 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:27 smithi189 ceph-mon[36179]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:27.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:27 smithi037 ceph-mon[29191]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:55:27.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:27 smithi037 ceph-mon[29191]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 -- 172.21.15.189:0/3656425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398100d60 msgr2=0x7fa398103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 --2- 172.21.15.189:0/3656425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398100d60 0x7fa398103140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa388009910 tx=0x7fa388009c20 comp rx=0 tx=0).stop 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 -- 172.21.15.189:0/3656425297 shutdown_connections 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 --2- 172.21.15.189:0/3656425297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa398103680 0x7fa398105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 --2- 172.21.15.189:0/3656425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398100d60 0x7fa398103140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 -- 172.21.15.189:0/3656425297 >> 172.21.15.189:0/3656425297 conn(0x7fa3980fa780 msgr2=0x7fa3980fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 -- 172.21.15.189:0/3656425297 shutdown_connections 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.456+0000 7fa39c884700 1 -- 172.21.15.189:0/3656425297 wait complete. 2024-04-15T22:55:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39c884700 1 Processor -- start 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39c884700 1 -- start start 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39c884700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa398100d60 0x7fa3981006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39c884700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 0x7fa3980fed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39c884700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980ff250 con 0x7fa398100d60 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39c884700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980ff390 con 0x7fa398103680 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa38ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 0x7fa3980fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa38ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 0x7fa3980fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58466/0 (socket says 172.21.15.189:58466) 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa38ffff700 1 -- 172.21.15.189:0/2765261494 learned_addr learned my addr 172.21.15.189:0/2765261494 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa39659c700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa398100d60 0x7fa3981006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa38ffff700 1 -- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa398100d60 msgr2=0x7fa3981006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa38ffff700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa398100d60 0x7fa3981006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.457+0000 7fa38ffff700 1 -- 172.21.15.189:0/2765261494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880095f0 con 0x7fa398103680 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.458+0000 7fa38ffff700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 0x7fa3980fed10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa38000d8d0 tx=0x7fa38000dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.458+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3800098a0 con 0x7fa398103680 2024-04-15T22:55:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.458+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa380010460 con 0x7fa398103680 2024-04-15T22:55:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.458+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa38000f5f0 con 0x7fa398103680 2024-04-15T22:55:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.458+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3980ff670 con 0x7fa398103680 2024-04-15T22:55:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.458+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3980ffb70 con 0x7fa398103680 2024-04-15T22:55:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.459+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39818fbc0 con 0x7fa398103680 2024-04-15T22:55:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.459+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa380009a00 con 0x7fa398103680 2024-04-15T22:55:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.460+0000 7fa38f7fe700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa38406c680 0x7fa38406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.460+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fa38008b270 con 0x7fa398103680 2024-04-15T22:55:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.460+0000 7fa39659c700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa38406c680 0x7fa38406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.461+0000 7fa39659c700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa38406c680 0x7fa38406eb30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa388006010 tx=0x7fa38800b540 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.464+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa3800598d0 con 0x7fa398103680 2024-04-15T22:55:29.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:29.661+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa398061190 con 0x7fa38406c680 2024-04-15T22:55:29.654 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:29.654 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.654 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:29.655 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:29 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:29.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:29 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:30.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:30 smithi189 ceph-mon[36179]: Detected new or changed devices on smithi189 2024-04-15T22:55:30.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:30 smithi189 ceph-mon[36179]: Adjusting osd_memory_target on smithi189 to 7456M 2024-04-15T22:55:30.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:55:30.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:55:30.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:30.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:30 smithi037 ceph-mon[29191]: Detected new or changed devices on smithi189 2024-04-15T22:55:30.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:30 smithi037 ceph-mon[29191]: Adjusting osd_memory_target on smithi189 to 7456M 2024-04-15T22:55:30.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:55:30.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:55:30.718 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:31.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:31 smithi189 ceph-mon[36179]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:55:31.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:31 smithi189 ceph-mon[36179]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:31.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:31 smithi037 ceph-mon[29191]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:55:31.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:31 smithi037 ceph-mon[29191]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:32.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:32 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/245246762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de638b57-8f16-4a8f-b0b0-bbc7d84831c2"}]: dispatch 2024-04-15T22:55:32.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:32 smithi189 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de638b57-8f16-4a8f-b0b0-bbc7d84831c2"}]: dispatch 2024-04-15T22:55:32.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:32 smithi189 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de638b57-8f16-4a8f-b0b0-bbc7d84831c2"}]': finished 2024-04-15T22:55:32.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:32 smithi189 ceph-mon[36179]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:55:32.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:32.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:32 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/245246762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de638b57-8f16-4a8f-b0b0-bbc7d84831c2"}]: dispatch 2024-04-15T22:55:32.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:32 smithi037 ceph-mon[29191]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de638b57-8f16-4a8f-b0b0-bbc7d84831c2"}]: dispatch 2024-04-15T22:55:32.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:32 smithi037 ceph-mon[29191]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de638b57-8f16-4a8f-b0b0-bbc7d84831c2"}]': finished 2024-04-15T22:55:32.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:32 smithi037 ceph-mon[29191]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:55:32.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:33.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:33 smithi189 ceph-mon[36179]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:33.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:33 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/3624458828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:55:33.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:33 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:33.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:33 smithi037 ceph-mon[29191]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:33.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:33 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/3624458828' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:55:33.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:33 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:35.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:35 smithi189 ceph-mon[36179]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:35.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:35 smithi037 ceph-mon[29191]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:37.611 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:37 smithi189 ceph-mon[36179]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:37.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:37 smithi037 ceph-mon[29191]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:39.630 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:39 smithi189 ceph-mon[36179]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:39.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:39 smithi037 ceph-mon[29191]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:40.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:40 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:55:40.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:40 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:40.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:40 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:55:40.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:40 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:41.689 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:41 smithi189 ceph-mon[36179]: Deploying daemon osd.5 on smithi189 2024-04-15T22:55:41.689 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:41 smithi189 ceph-mon[36179]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:41.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:41 smithi037 ceph-mon[29191]: Deploying daemon osd.5 on smithi189 2024-04-15T22:55:41.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:41 smithi037 ceph-mon[29191]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:43.430 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:43 smithi189 ceph-mon[36179]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:43.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:43 smithi037 ceph-mon[29191]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:45.463 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:45 smithi189 ceph-mon[36179]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:45.463 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:45.463 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:45.463 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:45 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:45.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:45 smithi037 ceph-mon[29191]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:45.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:45.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:45.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:45 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:47.463 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:47 smithi189 ceph-mon[36179]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:47.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:47 smithi037 ceph-mon[29191]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:48.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:48 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:48.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:48 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:55:48.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.877+0000 7fa38f7fe700 1 -- 172.21.15.189:0/2765261494 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa398061190 con 0x7fa38406c680 2024-04-15T22:55:48.870 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-04-15T22:55:48.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa38406c680 msgr2=0x7fa38406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:48.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa38406c680 0x7fa38406eb30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa388006010 tx=0x7fa38800b540 comp rx=0 tx=0).stop 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 msgr2=0x7fa3980fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 0x7fa3980fed10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa38000d8d0 tx=0x7fa38000dbe0 comp rx=0 tx=0).stop 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 shutdown_connections 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa38406c680 0x7fa38406eb30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa398100d60 0x7fa3981006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 --2- 172.21.15.189:0/2765261494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa398103680 0x7fa3980fed10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 >> 172.21.15.189:0/2765261494 conn(0x7fa3980fa780 msgr2=0x7fa3980fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 shutdown_connections 2024-04-15T22:55:48.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:55:48.880+0000 7fa39c884700 1 -- 172.21.15.189:0/2765261494 wait complete. 2024-04-15T22:55:49.526 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:49 smithi189 ceph-mon[36179]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:49.526 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:49.526 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:49.526 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:49 smithi189 ceph-mon[36179]: from='osd.5 [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:55:49.526 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:49 smithi189 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:55:49.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:49 smithi037 ceph-mon[29191]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:49.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:49.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:49.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:49 smithi037 ceph-mon[29191]: from='osd.5 [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:55:49.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:49 smithi037 ceph-mon[29191]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:55:49.837 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968@osd.5.service 2024-04-15T22:55:49.839 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-15T22:55:49.840 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-15T22:55:50.068 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:55:51.174 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:55:51.174 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:55:51.174 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:51.174 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: from='osd.5 [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:51.174 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:51.175 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:51.175 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:51.175 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:50 smithi037 ceph-mon[29191]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: from='osd.5 [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:51.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-mon[36179]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:51.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:55:50.869+0000 7f173cc41700 -1 osd.5 0 waiting for initial osdmap 2024-04-15T22:55:51.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:55:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:55:50.877+0000 7f173522f700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:55:51.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 -- 172.21.15.37:0/3238478676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fee50 msgr2=0x7f635c101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:51.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 --2- 172.21.15.37:0/3238478676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fee50 0x7f635c101580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6344009910 tx=0x7f6344009c20 comp rx=0 tx=0).stop 2024-04-15T22:55:51.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 -- 172.21.15.37:0/3238478676 shutdown_connections 2024-04-15T22:55:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 --2- 172.21.15.37:0/3238478676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fee50 0x7f635c101580 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 --2- 172.21.15.37:0/3238478676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635c0fe500 0x7f635c0fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 -- 172.21.15.37:0/3238478676 >> 172.21.15.37:0/3238478676 conn(0x7f635c0f9f10 msgr2=0x7f635c0fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 -- 172.21.15.37:0/3238478676 shutdown_connections 2024-04-15T22:55:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.790+0000 7f636200c700 1 -- 172.21.15.37:0/3238478676 wait complete. 2024-04-15T22:55:51.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f636200c700 1 Processor -- start 2024-04-15T22:55:51.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f636200c700 1 -- start start 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f636200c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 0x7f635c195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f636200c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635c0fee50 0x7f635c196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f636200c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c196690 con 0x7f635c0fe500 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f636200c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1967d0 con 0x7f635c0fee50 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 0x7f635c195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 0x7f635c195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34572/0 (socket says 172.21.15.37:34572) 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635b7fe700 1 -- 172.21.15.37:0/143118600 learned_addr learned my addr 172.21.15.37:0/143118600 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635affd700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635c0fee50 0x7f635c196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635b7fe700 1 -- 172.21.15.37:0/143118600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635c0fee50 msgr2=0x7f635c196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635b7fe700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635c0fee50 0x7f635c196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.791+0000 7f635b7fe700 1 -- 172.21.15.37:0/143118600 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63440095f0 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.792+0000 7f635b7fe700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 0x7f635c195b30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f634c00ef10 tx=0x7f634c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.792+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634c00c9f0 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.792+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f634c00f040 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.792+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c19b220 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.792+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634c0146b0 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.792+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c19b770 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.793+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c108d10 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.793+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f634c0078b0 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.794+0000 7f6358ff9700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f634806c680 0x7f634806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.794+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f634c08b860 con 0x7f635c0fe500 2024-04-15T22:55:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.794+0000 7f635affd700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f634806c680 0x7f634806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:51.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.794+0000 7f635affd700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f634806c680 0x7f634806eb30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63440059d0 tx=0x7f6344005940 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:51.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:51.799+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f634c059d80 con 0x7f635c0fe500 2024-04-15T22:55:51.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.001+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f635c108ed0 con 0x7f635c0fe500 2024-04-15T22:55:51.995 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:51 smithi037 ceph-mon[29191]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-15T22:55:51.995 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:51 smithi037 ceph-mon[29191]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:55:51.995 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:51.995 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:51 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:51.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.001+0000 7f6358ff9700 1 -- 172.21.15.37:0/143118600 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f634c059910 con 0x7f635c0fe500 2024-04-15T22:55:51.996 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f634806c680 msgr2=0x7f634806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f634806c680 0x7f634806eb30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63440059d0 tx=0x7f6344005940 comp rx=0 tx=0).stop 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 msgr2=0x7f635c195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 0x7f635c195b30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f634c00ef10 tx=0x7f634c00c3a0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 shutdown_connections 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f634806c680 0x7f634806eb30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f635c0fe500 0x7f635c195b30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 --2- 172.21.15.37:0/143118600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635c0fee50 0x7f635c196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:51.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 >> 172.21.15.37:0/143118600 conn(0x7f635c0f9f10 msgr2=0x7f635c1047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:51.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 shutdown_connections 2024-04-15T22:55:51.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:52.003+0000 7f636200c700 1 -- 172.21.15.37:0/143118600 wait complete. 2024-04-15T22:55:52.058 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:51 smithi189 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-15T22:55:52.058 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:51 smithi189 ceph-mon[36179]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:55:52.058 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:52.058 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:51 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:52.630 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1713221751,"num_in_osds":6,"osd_in_since":1713221732,"num_remapped_pgs":0} 2024-04-15T22:55:52.631 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:55:52.846 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:55:53.143 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: purged_snaps scrub starts 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: purged_snaps scrub ok 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: osd.5 [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] boot 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/143118600' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:55:53.144 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:52 smithi037 ceph-mon[29191]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: purged_snaps scrub starts 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: purged_snaps scrub ok 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: osd.5 [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] boot 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/143118600' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:55:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:52 smithi189 ceph-mon[36179]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:55:53.948 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:55:53.948 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:53.948 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:53.949 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:53 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:54.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:54.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:55:54.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:55:54.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:55:54.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:53 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 -- 172.21.15.37:0/394285749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e8075e80 msgr2=0x7f94e80ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/394285749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e8075e80 0x7f94e80ff350 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009940 tx=0x7f94d8009c50 comp rx=0 tx=0).stop 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 -- 172.21.15.37:0/394285749 shutdown_connections 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/394285749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e80ff890 0x7f94e80ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/394285749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e8075e80 0x7f94e80ff350 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 -- 172.21.15.37:0/394285749 >> 172.21.15.37:0/394285749 conn(0x7f94e80748b0 msgr2=0x7f94e8074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 -- 172.21.15.37:0/394285749 shutdown_connections 2024-04-15T22:55:54.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.444+0000 7f94ee7b9700 1 -- 172.21.15.37:0/394285749 wait complete. 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94ee7b9700 1 Processor -- start 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94ee7b9700 1 -- start start 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94ee7b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8075e80 0x7f94e8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94ee7b9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 0x7f94e8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94ee7b9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8198bf0 con 0x7f94e80ff890 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94ee7b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e819c980 con 0x7f94e8075e80 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94dffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 0x7f94e8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94dffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 0x7f94e8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34592/0 (socket says 172.21.15.37:34592) 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94dffff700 1 -- 172.21.15.37:0/1833954864 learned_addr learned my addr 172.21.15.37:0/1833954864 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94e7fff700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8075e80 0x7f94e8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94dffff700 1 -- 172.21.15.37:0/1833954864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8075e80 msgr2=0x7f94e8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94dffff700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8075e80 0x7f94e8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.445+0000 7f94dffff700 1 -- 172.21.15.37:0/1833954864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80095f0 con 0x7f94e80ff890 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.446+0000 7f94dffff700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 0x7f94e8198510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f94d000d8d0 tx=0x7f94d000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:54.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.446+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94d00098a0 con 0x7f94e80ff890 2024-04-15T22:55:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.446+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f94d0010460 con 0x7f94e80ff890 2024-04-15T22:55:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.446+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e819cc60 con 0x7f94e80ff890 2024-04-15T22:55:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.446+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94d000f5f0 con 0x7f94e80ff890 2024-04-15T22:55:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.446+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e819d1b0 con 0x7f94e80ff890 2024-04-15T22:55:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.447+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e804f2a0 con 0x7f94e80ff890 2024-04-15T22:55:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.447+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f94d0009a00 con 0x7f94e80ff890 2024-04-15T22:55:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.448+0000 7f94e5ffb700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f94d406c680 0x7f94d406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.448+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f94d008b420 con 0x7f94e80ff890 2024-04-15T22:55:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.448+0000 7f94e7fff700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f94d406c680 0x7f94d406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.448+0000 7f94e7fff700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f94d406c680 0x7f94d406eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f94d8006010 tx=0x7f94d8005a90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.453+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f94d0055e40 con 0x7f94e80ff890 2024-04-15T22:55:54.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.651+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94e819d4d0 con 0x7f94e80ff890 2024-04-15T22:55:54.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.652+0000 7f94e5ffb700 1 -- 172.21.15.37:0/1833954864 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11315 (secure 0 0 0) 0x7f94d0059460 con 0x7f94e80ff890 2024-04-15T22:55:54.646 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:55:54.646 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":40,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","created":"2024-04-15T22:49:47.691565+0000","modified":"2024-04-15T22:55:52.869487+0000","last_up_change":"2024-04-15T22:55:51.865136+0000","last_in_change":"2024-04-15T22:55:32.171674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:54:34.664768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc1abff5-8aea-47ca-982a-29cd590fc31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6803","nonce":2814326386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6805","nonce":2814326386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6809","nonce":2814326386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6807","nonce":2814326386}]},"public_addr":"172.21.15.37:6803/2814326386","cluster_addr":"172.21.15.37:6805/2814326386","heartbeat_back_addr":"172.21.15.37:6809/2814326386","heartbeat_front_addr":"172.21.15.37:6807/2814326386","state":["exists","up"]},{"osd":1,"uuid":"e875b0b4-8220-4f8a-9d7b-c2dffdd988dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6811","nonce":1887622200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6813","nonce":1887622200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6817","nonce":1887622200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6815","nonce":1887622200}]},"public_addr":"172.21.15.37:6811/1887622200","cluster_addr":"172.21.15.37:6813/1887622200","heartbeat_back_addr":"172.21.15.37:6817/1887622200","heartbeat_front_addr":"172.21.15.37:6815/1887622200","state":["exists","up"]},{"osd":2,"uuid":"08257888-e768-4845-9efa-8358f6e8caec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6819","nonce":503233976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6821","nonce":503233976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6825","nonce":503233976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6823","nonce":503233976}]},"public_addr":"172.21.15.37:6819/503233976","cluster_addr":"172.21.15.37:6821/503233976","heartbeat_back_addr":"172.21.15.37:6825/503233976","heartbeat_front_addr":"172.21.15.37:6823/503233976","state":["exists","up"]},{"osd":3,"uuid":"1d49e4a2-0131-436b-b02c-77dd4237bee8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6801","nonce":1840940998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6803","nonce":1840940998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6807","nonce":1840940998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6805","nonce":1840940998}]},"public_addr":"172.21.15.189:6801/1840940998","cluster_addr":"172.21.15.189:6803/1840940998","heartbeat_back_addr":"172.21.15.189:6807/1840940998","heartbeat_front_addr":"172.21.15.189:6805/1840940998","state":["exists","up"]},{"osd":4,"uuid":"b93aee26-d0dd-4cbc-b45e-b005033c1a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6809","nonce":2518250570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6811","nonce":2518250570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6815","nonce":2518250570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6813","nonce":2518250570}]},"public_addr":"172.21.15.189:6809/2518250570","cluster_addr":"172.21.15.189:6811/2518250570","heartbeat_back_addr":"172.21.15.189:6815/2518250570","heartbeat_front_addr":"172.21.15.189:6813/2518250570","state":["exists","up"]},{"osd":5,"uuid":"de638b57-8f16-4a8f-b0b0-bbc7d84831c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6817","nonce":580752685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6819","nonce":580752685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6823","nonce":580752685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6821","nonce":580752685}]},"public_addr":"172.21.15.189:6817/580752685","cluster_addr":"172.21.15.189:6819/580752685","heartbeat_back_addr":"172.21.15.189:6823/580752685","heartbeat_front_addr":"172.21.15.189:6821/580752685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:53:35.515515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:02.780098+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:31.718613+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:56.852313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:23.477486+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:50.019389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3357942121":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/1434563686":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3190201420":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/8957815":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6800/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/957260578":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/3560860596":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/1091007617":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6801/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/2768618180":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/3522001690":"2024-04-16T22:50:53.072525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f94d406c680 msgr2=0x7f94d406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f94d406c680 0x7f94d406eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f94d8006010 tx=0x7f94d8005a90 comp rx=0 tx=0).stop 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 msgr2=0x7f94e8198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 0x7f94e8198510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f94d000d8d0 tx=0x7f94d000dbe0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 shutdown_connections 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f94d406c680 0x7f94d406eb30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94e8075e80 0x7f94e8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 --2- 172.21.15.37:0/1833954864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94e80ff890 0x7f94e8198510 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:54.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 >> 172.21.15.37:0/1833954864 conn(0x7f94e80748b0 msgr2=0x7f94e80fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:54.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 shutdown_connections 2024-04-15T22:55:54.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:54.654+0000 7f94ee7b9700 1 -- 172.21.15.37:0/1833954864 wait complete. 2024-04-15T22:55:54.921 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:54 smithi037 ceph-mon[29191]: Detected new or changed devices on smithi189 2024-04-15T22:55:54.921 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:54 smithi037 ceph-mon[29191]: Adjusting osd_memory_target on smithi189 to 4970M 2024-04-15T22:55:54.921 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:54 smithi037 ceph-mon[29191]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:55:54.921 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:54 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1833954864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:55:55.132 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:54 smithi189 ceph-mon[36179]: Detected new or changed devices on smithi189 2024-04-15T22:55:55.133 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:54 smithi189 ceph-mon[36179]: Adjusting osd_memory_target on smithi189 to 4970M 2024-04-15T22:55:55.133 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:54 smithi189 ceph-mon[36179]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:55:55.133 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:54 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1833954864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:55:55.363 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T22:54:34.664768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T22:55:55.364 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-15T22:55:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:55:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.412+0000 7f20edd2a700 1 -- 172.21.15.37:0/625669129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ff3c0 msgr2=0x7f20e80ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.412+0000 7f20edd2a700 1 --2- 172.21.15.37:0/625669129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ff3c0 0x7f20e80ff790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009910 tx=0x7f20d8009c20 comp rx=0 tx=0).stop 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 -- 172.21.15.37:0/625669129 shutdown_connections 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 --2- 172.21.15.37:0/625669129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20e80ffd60 0x7f20e8103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 --2- 172.21.15.37:0/625669129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ff3c0 0x7f20e80ff790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 -- 172.21.15.37:0/625669129 >> 172.21.15.37:0/625669129 conn(0x7f20e80fac80 msgr2=0x7f20e80fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 -- 172.21.15.37:0/625669129 shutdown_connections 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 -- 172.21.15.37:0/625669129 wait complete. 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 Processor -- start 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.413+0000 7f20edd2a700 1 -- start start 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20edd2a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20e80ff3c0 0x7f20e8100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20edd2a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 0x7f20e8101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20edd2a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e8101850 con 0x7f20e80ffd60 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20edd2a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e8101990 con 0x7f20e80ff3c0 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 0x7f20e8101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 0x7f20e8101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34604/0 (socket says 172.21.15.37:34604) 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 -- 172.21.15.37:0/2110249763 learned_addr learned my addr 172.21.15.37:0/2110249763 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e77fe700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20e80ff3c0 0x7f20e8100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 -- 172.21.15.37:0/2110249763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20e80ff3c0 msgr2=0x7f20e8100dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20e80ff3c0 0x7f20e8100dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 -- 172.21.15.37:0/2110249763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d80095f0 con 0x7f20e80ffd60 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e6ffd700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 0x7f20e8101310 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00c960 tx=0x7f20dc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20dc007a70 con 0x7f20e80ffd60 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f20dc007bd0 con 0x7f20e80ffd60 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e81a20f0 con 0x7f20e80ffd60 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20dc018630 con 0x7f20e80ffd60 2024-04-15T22:55:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.414+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e81a2450 con 0x7f20e80ffd60 2024-04-15T22:55:57.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.415+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e81086e0 con 0x7f20e80ffd60 2024-04-15T22:55:57.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.416+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f20dc00f450 con 0x7f20e80ffd60 2024-04-15T22:55:57.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.416+0000 7f20e4ff9700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f20d006c680 0x7f20d006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:55:57.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.417+0000 7f20e77fe700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f20d006c680 0x7f20d006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:55:57.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.417+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f20dc08b4e0 con 0x7f20e80ffd60 2024-04-15T22:55:57.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.417+0000 7f20e77fe700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f20d006c680 0x7f20d006eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20d8006010 tx=0x7f20d801a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:55:57.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.421+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f20dc055f00 con 0x7f20e80ffd60 2024-04-15T22:55:57.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:57 smithi037 ceph-mon[29191]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:55:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.618+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20e8101b50 con 0x7f20e80ffd60 2024-04-15T22:55:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.618+0000 7f20e4ff9700 1 -- 172.21.15.37:0/2110249763 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20dc059520 con 0x7f20e80ffd60 2024-04-15T22:55:57.611 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f20d006c680 msgr2=0x7f20d006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f20d006c680 0x7f20d006eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20d8006010 tx=0x7f20d801a040 comp rx=0 tx=0).stop 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 msgr2=0x7f20e8101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 0x7f20e8101310 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00c960 tx=0x7f20dc00cc70 comp rx=0 tx=0).stop 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 shutdown_connections 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f20d006c680 0x7f20d006eb30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20e80ff3c0 0x7f20e8100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 --2- 172.21.15.37:0/2110249763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20e80ffd60 0x7f20e8101310 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:55:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 >> 172.21.15.37:0/2110249763 conn(0x7f20e80fac80 msgr2=0x7f20e8069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:55:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 shutdown_connections 2024-04-15T22:55:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:55:57.620+0000 7f20edd2a700 1 -- 172.21.15.37:0/2110249763 wait complete. 2024-04-15T22:55:57.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:57 smithi189 ceph-mon[36179]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:55:58.296 INFO:tasks.cephadm:Setting up client nodes... 2024-04-15T22:55:58.297 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T22:55:58.517 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:55:58.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:58 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2110249763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:55:58.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:58 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2110249763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:55:59.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:55:59 smithi189 ceph-mon[36179]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:55:59.755 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:55:59 smithi037 ceph-mon[29191]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 -- 172.21.15.37:0/1656493678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 msgr2=0x7fe270105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 --2- 172.21.15.37:0/1656493678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270105a70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe264009910 tx=0x7fe264009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 -- 172.21.15.37:0/1656493678 shutdown_connections 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 --2- 172.21.15.37:0/1656493678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270105a70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 --2- 172.21.15.37:0/1656493678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270100d70 0x7fe270103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 -- 172.21.15.37:0/1656493678 >> 172.21.15.37:0/1656493678 conn(0x7fe2700fa6d0 msgr2=0x7fe2700fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 -- 172.21.15.37:0/1656493678 shutdown_connections 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.128+0000 7fe277551700 1 -- 172.21.15.37:0/1656493678 wait complete. 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe277551700 1 Processor -- start 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe277551700 1 -- start start 2024-04-15T22:56:00.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe277551700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270100d70 0x7fe270195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe277551700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe277551700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe270196600 con 0x7fe270103690 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe277551700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe270196740 con 0x7fe270100d70 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe274aec700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe274aec700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57452/0 (socket says 172.21.15.37:57452) 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe274aec700 1 -- 172.21.15.37:0/3271949685 learned_addr learned my addr 172.21.15.37:0/3271949685 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe2752ed700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270100d70 0x7fe270195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe274aec700 1 -- 172.21.15.37:0/3271949685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270100d70 msgr2=0x7fe270195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe274aec700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270100d70 0x7fe270195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.129+0000 7fe274aec700 1 -- 172.21.15.37:0/3271949685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2640095f0 con 0x7fe270103690 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.130+0000 7fe274aec700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270195fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe26400b5c0 tx=0x7fe26400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.130+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe26401d070 con 0x7fe270103690 2024-04-15T22:56:00.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.130+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe26400f460 con 0x7fe270103690 2024-04-15T22:56:00.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.130+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27019b190 con 0x7fe270103690 2024-04-15T22:56:00.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.130+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe264021640 con 0x7fe270103690 2024-04-15T22:56:00.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.130+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27019b680 con 0x7fe270103690 2024-04-15T22:56:00.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.131+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27018fc70 con 0x7fe270103690 2024-04-15T22:56:00.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.131+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe26400fa60 con 0x7fe270103690 2024-04-15T22:56:00.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.132+0000 7fe26e7fc700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe25806c680 0x7fe25806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:00.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.132+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe26408c790 con 0x7fe270103690 2024-04-15T22:56:00.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.132+0000 7fe2752ed700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe25806c680 0x7fe25806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:00.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.132+0000 7fe2752ed700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe25806c680 0x7fe25806eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe260009550 tx=0x7fe2600091b0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:00.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.136+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe26405ac40 con 0x7fe270103690 2024-04-15T22:56:00.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.419+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe270061960 con 0x7fe270103690 2024-04-15T22:56:00.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.423+0000 7fe26e7fc700 1 -- 172.21.15.37:0/3271949685 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe26405a7d0 con 0x7fe270103690 2024-04-15T22:56:00.416 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-04-15T22:56:00.416 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCAsB1mqvcOGRAAvLoFmoJaOyzJ858L2vEd7A== 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe25806c680 msgr2=0x7fe25806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe25806c680 0x7fe25806eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe260009550 tx=0x7fe2600091b0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 msgr2=0x7fe270195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270195fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe26400b5c0 tx=0x7fe26400bd70 comp rx=0 tx=0).stop 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 shutdown_connections 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe25806c680 0x7fe25806eb30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270100d70 0x7fe270195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 --2- 172.21.15.37:0/3271949685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe270103690 0x7fe270195fe0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:00.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 >> 172.21.15.37:0/3271949685 conn(0x7fe2700fa6d0 msgr2=0x7fe2700fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:00.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 shutdown_connections 2024-04-15T22:56:00.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:00.425+0000 7fe277551700 1 -- 172.21.15.37:0/3271949685 wait complete. 2024-04-15T22:56:00.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:00 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3271949685' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:56:00.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:00 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3271949685' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:56:00.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:00 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3271949685' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:56:00.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:00 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3271949685' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:56:01.199 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:56:01.199 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-15T22:56:01.199 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T22:56:01.247 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T22:56:01.469 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi189/config 2024-04-15T22:56:01.590 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:01 smithi189 ceph-mon[36179]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:01.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:01 smithi037 ceph-mon[29191]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:03.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 -- 172.21.15.189:0/103729698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 msgr2=0x7f3b180ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:03.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/103729698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b180ff560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009910 tx=0x7f3b0c009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:03.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 -- 172.21.15.189:0/103729698 shutdown_connections 2024-04-15T22:56:03.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/103729698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18100350 0x7f3b181007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:03.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/103729698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b180ff560 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:03.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 -- 172.21.15.189:0/103729698 >> 172.21.15.189:0/103729698 conn(0x7f3b180fa700 msgr2=0x7f3b180fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 -- 172.21.15.189:0/103729698 shutdown_connections 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.132+0000 7f3b1f21e700 1 -- 172.21.15.189:0/103729698 wait complete. 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1f21e700 1 Processor -- start 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1f21e700 1 -- start start 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1f21e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b18194a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1f21e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18100350 0x7f3b18194fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1f21e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b181955f0 con 0x7f3b18100350 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1f21e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b18195730 con 0x7f3b180ff150 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b18194a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b18194a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47646/0 (socket says 172.21.15.189:47646) 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 -- 172.21.15.189:0/2315063820 learned_addr learned my addr 172.21.15.189:0/2315063820 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b17fff700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18100350 0x7f3b18194fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:03.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 -- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18100350 msgr2=0x7f3b18194fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18100350 0x7f3b18194fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 -- 172.21.15.189:0/2315063820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c0095f0 con 0x7f3b180ff150 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.133+0000 7f3b1cfba700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b18194a90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00bda0 tx=0x7f3b0c00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.134+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0c01f070 con 0x7f3b180ff150 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.134+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b0c004c50 con 0x7f3b180ff150 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.134+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0c0247c0 con 0x7f3b180ff150 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.134+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b18068dd0 con 0x7f3b180ff150 2024-04-15T22:56:03.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.134+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b180692c0 con 0x7f3b180ff150 2024-04-15T22:56:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.135+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b18107d20 con 0x7f3b180ff150 2024-04-15T22:56:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.135+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3b0c019b50 con 0x7f3b180ff150 2024-04-15T22:56:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.136+0000 7f3b15ffb700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0806c680 0x7f3b0806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.136+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3b0c08ee30 con 0x7f3b180ff150 2024-04-15T22:56:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.136+0000 7f3b17fff700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0806c680 0x7f3b0806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.137+0000 7f3b17fff700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0806c680 0x7f3b0806eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b04006fd0 tx=0x7f3b04008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.140+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3b0c05d360 con 0x7f3b180ff150 2024-04-15T22:56:03.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.408+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f3b18069cc0 con 0x7f3b180ff150 2024-04-15T22:56:03.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.413+0000 7f3b15ffb700 1 -- 172.21.15.189:0/2315063820 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f3b0c029050 con 0x7f3b180ff150 2024-04-15T22:56:03.407 INFO:teuthology.orchestra.run.smithi189.stdout:[client.1] 2024-04-15T22:56:03.407 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCDsB1mEGSDGBAA0rkyOxeoCCVQeLt5uQ4kfA== 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0806c680 msgr2=0x7f3b0806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0806c680 0x7f3b0806eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b04006fd0 tx=0x7f3b04008040 comp rx=0 tx=0).stop 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 msgr2=0x7f3b18194a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b18194a90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00bda0 tx=0x7f3b0c00be80 comp rx=0 tx=0).stop 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 shutdown_connections 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0806c680 0x7f3b0806eb30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180ff150 0x7f3b18194a90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:03.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 --2- 172.21.15.189:0/2315063820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18100350 0x7f3b18194fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:03.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 >> 172.21.15.189:0/2315063820 conn(0x7f3b180fa700 msgr2=0x7f3b18103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:03.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.415+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 shutdown_connections 2024-04-15T22:56:03.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:56:03.416+0000 7f3b1f21e700 1 -- 172.21.15.189:0/2315063820 wait complete. 2024-04-15T22:56:03.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:03 smithi037 ceph-mon[29191]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:03.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:03 smithi189 ceph-mon[36179]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:04.008 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:56:04.008 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-15T22:56:04.008 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T22:56:04.054 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T22:56:04.054 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-15T22:56:04.054 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-15T22:56:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:04.670 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:04.670 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:04 smithi037 ceph-mon[29191]: from='client.? 172.21.15.189:0/2315063820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:56:04.670 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:04 smithi037 ceph-mon[29191]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:56:04.670 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:04 smithi037 ceph-mon[29191]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:56:04.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:04.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:04 smithi189 ceph-mon[36179]: from='client.? 172.21.15.189:0/2315063820' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:56:04.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:04 smithi189 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:56:04.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:04 smithi189 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:56:05.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:05 smithi189 ceph-mon[36179]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:05.726 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:05 smithi037 ceph-mon[29191]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 -- 172.21.15.37:0/1720706256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ff3c0 msgr2=0x7ff8200ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 --2- 172.21.15.37:0/1720706256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ff3c0 0x7ff8200ff790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff810009910 tx=0x7ff810009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 -- 172.21.15.37:0/1720706256 shutdown_connections 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 --2- 172.21.15.37:0/1720706256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8200ffd60 0x7ff820103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 --2- 172.21.15.37:0/1720706256 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ff3c0 0x7ff8200ff790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 -- 172.21.15.37:0/1720706256 >> 172.21.15.37:0/1720706256 conn(0x7ff8200fac80 msgr2=0x7ff8200fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 -- 172.21.15.37:0/1720706256 shutdown_connections 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 -- 172.21.15.37:0/1720706256 wait complete. 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.091+0000 7ff825e73700 1 Processor -- start 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff825e73700 1 -- start start 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff825e73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8200ff3c0 0x7ff820195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff825e73700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 0x7ff8201964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff825e73700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820196ba0 con 0x7ff8200ffd60 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff825e73700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82019a930 con 0x7ff8200ff3c0 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 0x7ff8201964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 0x7ff8201964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57464/0 (socket says 172.21.15.37:57464) 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 -- 172.21.15.37:0/3556728527 learned_addr learned my addr 172.21.15.37:0/3556728527 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81f7fe700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8200ff3c0 0x7ff820195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 -- 172.21.15.37:0/3556728527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8200ff3c0 msgr2=0x7ff820195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8200ff3c0 0x7ff820195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 -- 172.21.15.37:0/3556728527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8100095f0 con 0x7ff8200ffd60 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.092+0000 7ff81effd700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 0x7ff8201964c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff81400b480 tx=0x7ff81400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.093+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff814011800 con 0x7ff8200ffd60 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.093+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff814011e10 con 0x7ff8200ffd60 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.093+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82019ac10 con 0x7ff8200ffd60 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.093+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff814010ea0 con 0x7ff8200ffd60 2024-04-15T22:56:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.093+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82019b160 con 0x7ff8200ffd60 2024-04-15T22:56:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.094+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8201086e0 con 0x7ff8200ffd60 2024-04-15T22:56:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.094+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff814010380 con 0x7ff8200ffd60 2024-04-15T22:56:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.094+0000 7ff81cff9700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff80806c680 0x7ff80806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.095+0000 7ff81f7fe700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff80806c680 0x7ff80806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.095+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff81408b0b0 con 0x7ff8200ffd60 2024-04-15T22:56:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.095+0000 7ff81f7fe700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff80806c680 0x7ff80806eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff810006010 tx=0x7ff81001a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.099+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff814055ad0 con 0x7ff8200ffd60 2024-04-15T22:56:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.370+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff82019b3e0 con 0x7ff8200ffd60 2024-04-15T22:56:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.373+0000 7ff81cff9700 1 -- 172.21.15.37:0/3556728527 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+172852 (secure 0 0 0) 0x7ff8140590f0 con 0x7ff8200ffd60 2024-04-15T22:56:06.368 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:56:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.375+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff80806c680 msgr2=0x7ff80806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff80806c680 0x7ff80806eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff810006010 tx=0x7ff81001a040 comp rx=0 tx=0).stop 2024-04-15T22:56:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 msgr2=0x7ff8201964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 0x7ff8201964c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff81400b480 tx=0x7ff81400b790 comp rx=0 tx=0).stop 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 shutdown_connections 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff80806c680 0x7ff80806eb30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8200ff3c0 0x7ff820195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 --2- 172.21.15.37:0/3556728527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8200ffd60 0x7ff8201964c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 >> 172.21.15.37:0/3556728527 conn(0x7ff8200fac80 msgr2=0x7ff8200690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 shutdown_connections 2024-04-15T22:56:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:06.376+0000 7ff825e73700 1 -- 172.21.15.37:0/3556728527 wait complete. 2024-04-15T22:56:06.688 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:06 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3556728527' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:56:06.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:06 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3556728527' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:56:06.995 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":19,"active_gid":14223,"active_name":"smithi037.gtwkck","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":935132805},{"type":"v1","addr":"172.21.15.37:6801","nonce":935132805}]},"active_addr":"172.21.15.37:6801/935132805","active_change":"2024-04-15T22:52:18.338575+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14246,"name":"smithi189.lanqnp","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.37:8443/","prometheus":"http://172.21.15.37:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":2264143714}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":3924340867}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":3058772019}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":1688979906}]}]} 2024-04-15T22:56:06.999 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-15T22:56:06.999 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T22:56:06.999 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:56:07.213 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:07.617 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:07 smithi037 ceph-mon[29191]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:07.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:07 smithi189 ceph-mon[36179]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:08.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 -- 172.21.15.37:0/90789452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0ff8c0 msgr2=0x7f482c10c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:08.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 --2- 172.21.15.37:0/90789452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0ff8c0 0x7f482c10c620 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f481c009910 tx=0x7f481c009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:08.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 -- 172.21.15.37:0/90789452 shutdown_connections 2024-04-15T22:56:08.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 --2- 172.21.15.37:0/90789452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0ff8c0 0x7f482c10c620 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:08.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 --2- 172.21.15.37:0/90789452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f482c0fefb0 0x7f482c0ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:08.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 -- 172.21.15.37:0/90789452 >> 172.21.15.37:0/90789452 conn(0x7f482c0fac80 msgr2=0x7f482c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:08.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 -- 172.21.15.37:0/90789452 shutdown_connections 2024-04-15T22:56:08.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 -- 172.21.15.37:0/90789452 wait complete. 2024-04-15T22:56:08.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.855+0000 7f4834286700 1 Processor -- start 2024-04-15T22:56:08.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4834286700 1 -- start start 2024-04-15T22:56:08.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4834286700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 0x7f482c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4834286700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f482c0ff8c0 0x7f482c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4834286700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c198c60 con 0x7f482c0fefb0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4834286700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c19c9f0 con 0x7f482c0ff8c0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 0x7f482c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 0x7f482c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57480/0 (socket says 172.21.15.37:57480) 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 -- 172.21.15.37:0/756652757 learned_addr learned my addr 172.21.15.37:0/756652757 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4831821700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f482c0ff8c0 0x7f482c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 -- 172.21.15.37:0/756652757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f482c0ff8c0 msgr2=0x7f482c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f482c0ff8c0 0x7f482c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 -- 172.21.15.37:0/756652757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0095f0 con 0x7f482c0fefb0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f4832022700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 0x7f482c198040 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f482800d8d0 tx=0x7f482800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.856+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4828009940 con 0x7f482c0fefb0 2024-04-15T22:56:08.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.857+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c19ccd0 con 0x7f482c0fefb0 2024-04-15T22:56:08.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.857+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4828010460 con 0x7f482c0fefb0 2024-04-15T22:56:08.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.857+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f482800f5f0 con 0x7f482c0fefb0 2024-04-15T22:56:08.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.857+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c19d220 con 0x7f482c0fefb0 2024-04-15T22:56:08.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.858+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c109d90 con 0x7f482c0fefb0 2024-04-15T22:56:08.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.858+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f48280105d0 con 0x7f482c0fefb0 2024-04-15T22:56:08.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.859+0000 7f48237fe700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f481806c680 0x7f481806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:08.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.859+0000 7f4831821700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f481806c680 0x7f481806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:08.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.859+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f482808b540 con 0x7f482c0fefb0 2024-04-15T22:56:08.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.859+0000 7f4831821700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f481806c680 0x7f481806eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f481c005b40 tx=0x7f481c01a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:08.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:08.863+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4828055f60 con 0x7f482c0fefb0 2024-04-15T22:56:09.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.061+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f482c04f2a0 con 0x7f482c0fefb0 2024-04-15T22:56:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.062+0000 7f48237fe700 1 -- 172.21.15.37:0/756652757 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11315 (secure 0 0 0) 0x7f4828059580 con 0x7f482c0fefb0 2024-04-15T22:56:09.055 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:56:09.056 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":40,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","created":"2024-04-15T22:49:47.691565+0000","modified":"2024-04-15T22:55:52.869487+0000","last_up_change":"2024-04-15T22:55:51.865136+0000","last_in_change":"2024-04-15T22:55:32.171674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:54:34.664768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc1abff5-8aea-47ca-982a-29cd590fc31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6803","nonce":2814326386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6805","nonce":2814326386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6809","nonce":2814326386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6807","nonce":2814326386}]},"public_addr":"172.21.15.37:6803/2814326386","cluster_addr":"172.21.15.37:6805/2814326386","heartbeat_back_addr":"172.21.15.37:6809/2814326386","heartbeat_front_addr":"172.21.15.37:6807/2814326386","state":["exists","up"]},{"osd":1,"uuid":"e875b0b4-8220-4f8a-9d7b-c2dffdd988dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6811","nonce":1887622200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6813","nonce":1887622200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6817","nonce":1887622200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6815","nonce":1887622200}]},"public_addr":"172.21.15.37:6811/1887622200","cluster_addr":"172.21.15.37:6813/1887622200","heartbeat_back_addr":"172.21.15.37:6817/1887622200","heartbeat_front_addr":"172.21.15.37:6815/1887622200","state":["exists","up"]},{"osd":2,"uuid":"08257888-e768-4845-9efa-8358f6e8caec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6819","nonce":503233976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6821","nonce":503233976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6825","nonce":503233976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6823","nonce":503233976}]},"public_addr":"172.21.15.37:6819/503233976","cluster_addr":"172.21.15.37:6821/503233976","heartbeat_back_addr":"172.21.15.37:6825/503233976","heartbeat_front_addr":"172.21.15.37:6823/503233976","state":["exists","up"]},{"osd":3,"uuid":"1d49e4a2-0131-436b-b02c-77dd4237bee8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6801","nonce":1840940998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6803","nonce":1840940998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6807","nonce":1840940998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6805","nonce":1840940998}]},"public_addr":"172.21.15.189:6801/1840940998","cluster_addr":"172.21.15.189:6803/1840940998","heartbeat_back_addr":"172.21.15.189:6807/1840940998","heartbeat_front_addr":"172.21.15.189:6805/1840940998","state":["exists","up"]},{"osd":4,"uuid":"b93aee26-d0dd-4cbc-b45e-b005033c1a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6809","nonce":2518250570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6811","nonce":2518250570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6815","nonce":2518250570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6813","nonce":2518250570}]},"public_addr":"172.21.15.189:6809/2518250570","cluster_addr":"172.21.15.189:6811/2518250570","heartbeat_back_addr":"172.21.15.189:6815/2518250570","heartbeat_front_addr":"172.21.15.189:6813/2518250570","state":["exists","up"]},{"osd":5,"uuid":"de638b57-8f16-4a8f-b0b0-bbc7d84831c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6817","nonce":580752685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6819","nonce":580752685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6823","nonce":580752685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6821","nonce":580752685}]},"public_addr":"172.21.15.189:6817/580752685","cluster_addr":"172.21.15.189:6819/580752685","heartbeat_back_addr":"172.21.15.189:6823/580752685","heartbeat_front_addr":"172.21.15.189:6821/580752685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:53:35.515515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:02.780098+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:31.718613+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:56.852313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:23.477486+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:50.019389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3357942121":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/1434563686":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3190201420":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/8957815":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6800/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/957260578":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/3560860596":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/1091007617":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6801/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/2768618180":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/3522001690":"2024-04-16T22:50:53.072525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:56:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f481806c680 msgr2=0x7f481806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f481806c680 0x7f481806eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f481c005b40 tx=0x7f481c01a040 comp rx=0 tx=0).stop 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 msgr2=0x7f482c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 0x7f482c198040 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f482800d8d0 tx=0x7f482800dc90 comp rx=0 tx=0).stop 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 shutdown_connections 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f481806c680 0x7f481806eb30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f482c0fefb0 0x7f482c198040 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 --2- 172.21.15.37:0/756652757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f482c0ff8c0 0x7f482c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.064+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 >> 172.21.15.37:0/756652757 conn(0x7f482c0fac80 msgr2=0x7f482c0fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.065+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 shutdown_connections 2024-04-15T22:56:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:09.065+0000 7f4834286700 1 -- 172.21.15.37:0/756652757 wait complete. 2024-04-15T22:56:09.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:09 smithi037 ceph-mon[29191]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:09.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:09 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/756652757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:56:09.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:09 smithi189 ceph-mon[36179]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:09.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:09 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/756652757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:56:09.735 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T22:56:09.735 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:56:09.950 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:11.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 -- 172.21.15.37:0/958981601 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8518104130 msgr2=0x7f8518104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:11.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 --2- 172.21.15.37:0/958981601 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8518104130 0x7f8518104500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8508009910 tx=0x7f8508009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 -- 172.21.15.37:0/958981601 shutdown_connections 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 --2- 172.21.15.37:0/958981601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85180ffbd0 0x7f8518100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 --2- 172.21.15.37:0/958981601 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8518104130 0x7f8518104500 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 -- 172.21.15.37:0/958981601 >> 172.21.15.37:0/958981601 conn(0x7f8518075130 msgr2=0x7f8518075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 -- 172.21.15.37:0/958981601 shutdown_connections 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.541+0000 7f851ddb9700 1 -- 172.21.15.37:0/958981601 wait complete. 2024-04-15T22:56:11.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.542+0000 7f851ddb9700 1 Processor -- start 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.542+0000 7f851ddb9700 1 -- start start 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.542+0000 7f851ddb9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 0x7f8518198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.542+0000 7f851ddb9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8518104130 0x7f8518198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.542+0000 7f851ddb9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8518198c60 con 0x7f85180ffbd0 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.542+0000 7f851ddb9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851819c9f0 con 0x7f8518104130 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 0x7f8518198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 0x7f8518198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46518/0 (socket says 172.21.15.37:46518) 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 -- 172.21.15.37:0/1819151615 learned_addr learned my addr 172.21.15.37:0/1819151615 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f8516ffd700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8518104130 0x7f8518198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 -- 172.21.15.37:0/1819151615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8518104130 msgr2=0x7f8518198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8518104130 0x7f8518198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 -- 172.21.15.37:0/1819151615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85080095f0 con 0x7f85180ffbd0 2024-04-15T22:56:11.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f85177fe700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 0x7f8518198040 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f850800bd70 tx=0x7f850800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f850801d070 con 0x7f85180ffbd0 2024-04-15T22:56:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f850800f460 con 0x7f85180ffbd0 2024-04-15T22:56:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8508017630 con 0x7f85180ffbd0 2024-04-15T22:56:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851819cc70 con 0x7f85180ffbd0 2024-04-15T22:56:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.543+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851819d110 con 0x7f85180ffbd0 2024-04-15T22:56:11.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.544+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851804f2a0 con 0x7f85180ffbd0 2024-04-15T22:56:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.545+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f8508003a60 con 0x7f85180ffbd0 2024-04-15T22:56:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.545+0000 7f8514ff9700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f850006c680 0x7f850006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.545+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f850808c980 con 0x7f85180ffbd0 2024-04-15T22:56:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.545+0000 7f8516ffd700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f850006c680 0x7f850006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:11.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.546+0000 7f8516ffd700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f850006c680 0x7f850006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8518199660 tx=0x7f850c00b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:11.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.550+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f85080573a0 con 0x7f85180ffbd0 2024-04-15T22:56:11.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:11 smithi037 ceph-mon[29191]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:11.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:11 smithi189 ceph-mon[36179]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:11.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.760+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f851819d420 con 0x7f85180ffbd0 2024-04-15T22:56:11.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.761+0000 7f8514ff9700 1 -- 172.21.15.37:0/1819151615 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11315 (secure 0 0 0) 0x7f850805a9c0 con 0x7f85180ffbd0 2024-04-15T22:56:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:56:11.754 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":40,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","created":"2024-04-15T22:49:47.691565+0000","modified":"2024-04-15T22:55:52.869487+0000","last_up_change":"2024-04-15T22:55:51.865136+0000","last_in_change":"2024-04-15T22:55:32.171674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:54:34.664768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc1abff5-8aea-47ca-982a-29cd590fc31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6803","nonce":2814326386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6805","nonce":2814326386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6809","nonce":2814326386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6807","nonce":2814326386}]},"public_addr":"172.21.15.37:6803/2814326386","cluster_addr":"172.21.15.37:6805/2814326386","heartbeat_back_addr":"172.21.15.37:6809/2814326386","heartbeat_front_addr":"172.21.15.37:6807/2814326386","state":["exists","up"]},{"osd":1,"uuid":"e875b0b4-8220-4f8a-9d7b-c2dffdd988dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6811","nonce":1887622200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6813","nonce":1887622200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6817","nonce":1887622200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6815","nonce":1887622200}]},"public_addr":"172.21.15.37:6811/1887622200","cluster_addr":"172.21.15.37:6813/1887622200","heartbeat_back_addr":"172.21.15.37:6817/1887622200","heartbeat_front_addr":"172.21.15.37:6815/1887622200","state":["exists","up"]},{"osd":2,"uuid":"08257888-e768-4845-9efa-8358f6e8caec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6819","nonce":503233976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6821","nonce":503233976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6825","nonce":503233976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6823","nonce":503233976}]},"public_addr":"172.21.15.37:6819/503233976","cluster_addr":"172.21.15.37:6821/503233976","heartbeat_back_addr":"172.21.15.37:6825/503233976","heartbeat_front_addr":"172.21.15.37:6823/503233976","state":["exists","up"]},{"osd":3,"uuid":"1d49e4a2-0131-436b-b02c-77dd4237bee8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6801","nonce":1840940998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6803","nonce":1840940998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6807","nonce":1840940998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6805","nonce":1840940998}]},"public_addr":"172.21.15.189:6801/1840940998","cluster_addr":"172.21.15.189:6803/1840940998","heartbeat_back_addr":"172.21.15.189:6807/1840940998","heartbeat_front_addr":"172.21.15.189:6805/1840940998","state":["exists","up"]},{"osd":4,"uuid":"b93aee26-d0dd-4cbc-b45e-b005033c1a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6809","nonce":2518250570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6811","nonce":2518250570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6815","nonce":2518250570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6813","nonce":2518250570}]},"public_addr":"172.21.15.189:6809/2518250570","cluster_addr":"172.21.15.189:6811/2518250570","heartbeat_back_addr":"172.21.15.189:6815/2518250570","heartbeat_front_addr":"172.21.15.189:6813/2518250570","state":["exists","up"]},{"osd":5,"uuid":"de638b57-8f16-4a8f-b0b0-bbc7d84831c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6817","nonce":580752685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6819","nonce":580752685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6823","nonce":580752685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6821","nonce":580752685}]},"public_addr":"172.21.15.189:6817/580752685","cluster_addr":"172.21.15.189:6819/580752685","heartbeat_back_addr":"172.21.15.189:6823/580752685","heartbeat_front_addr":"172.21.15.189:6821/580752685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:53:35.515515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:02.780098+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:31.718613+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:56.852313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:23.477486+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:50.019389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3357942121":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/1434563686":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3190201420":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/8957815":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6800/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/957260578":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/3560860596":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/1091007617":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6801/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/2768618180":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/3522001690":"2024-04-16T22:50:53.072525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:56:11.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f850006c680 msgr2=0x7f850006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:11.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f850006c680 0x7f850006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8518199660 tx=0x7f850c00b410 comp rx=0 tx=0).stop 2024-04-15T22:56:11.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 msgr2=0x7f8518198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:11.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 0x7f8518198040 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f850800bd70 tx=0x7f850800bda0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 shutdown_connections 2024-04-15T22:56:11.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f850006c680 0x7f850006eb30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85180ffbd0 0x7f8518198040 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 --2- 172.21.15.37:0/1819151615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8518104130 0x7f8518198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:11.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.763+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 >> 172.21.15.37:0/1819151615 conn(0x7f8518075130 msgr2=0x7f8518068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:11.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.764+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 shutdown_connections 2024-04-15T22:56:11.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:11.764+0000 7f851ddb9700 1 -- 172.21.15.37:0/1819151615 wait complete. 2024-04-15T22:56:12.466 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-15T22:56:12.467 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-15T22:56:12.467 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-15T22:56:12.468 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-15T22:56:12.468 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-15T22:56:12.468 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-15T22:56:12.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:12 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1819151615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:56:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:12.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:12 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1819151615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:56:13.539 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:13 smithi037 ceph-mon[29191]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:13.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:13 smithi189 ceph-mon[36179]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:14.161 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:14.161 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:14.161 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:15.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:15 smithi189 ceph-mon[36179]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:15.759 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:15 smithi037 ceph-mon[29191]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:16.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 -- 172.21.15.37:0/571669652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ffd50 msgr2=0x7fa55c103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 --2- 172.21.15.37:0/571669652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ffd50 0x7fa55c103d20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa550009910 tx=0x7fa550009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 -- 172.21.15.37:0/571669652 shutdown_connections 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 --2- 172.21.15.37:0/571669652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ffd50 0x7fa55c103d20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 --2- 172.21.15.37:0/571669652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c0ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 -- 172.21.15.37:0/571669652 >> 172.21.15.37:0/571669652 conn(0x7fa55c0fac90 msgr2=0x7fa55c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 -- 172.21.15.37:0/571669652 shutdown_connections 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.988+0000 7fa560ca4700 1 -- 172.21.15.37:0/571669652 wait complete. 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.989+0000 7fa560ca4700 1 Processor -- start 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.989+0000 7fa560ca4700 1 -- start start 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.989+0000 7fa560ca4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa560ca4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55c0ffd50 0x7fa55c197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa560ca4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c197950 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa560ca4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c197a90 con 0x7fa55c0ffd50 2024-04-15T22:56:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46534/0 (socket says 172.21.15.37:46534) 2024-04-15T22:56:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 -- 172.21.15.37:0/69856140 learned_addr learned my addr 172.21.15.37:0/69856140 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 -- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55c0ffd50 msgr2=0x7fa55c197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa559d9b700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55c0ffd50 0x7fa55c197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55c0ffd50 0x7fa55c197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 -- 172.21.15.37:0/69856140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5500095f0 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa55a59c700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c196df0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00b4b0 tx=0x7fa54c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa54c011800 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c19fe60 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa54c011e10 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa54c00f2e0 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.990+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55c1a0320 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.991+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa55c10c560 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.992+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa54c010380 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.992+0000 7fa54b7fe700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa54406c680 0x7fa54406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.992+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa54c08b1a0 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.992+0000 7fa559d9b700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa54406c680 0x7fa54406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.993+0000 7fa54b7fe700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] conn(0x7fa544072350 0x7fa544074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.993+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 --> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa544074e10 con 0x7fa544072350 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.993+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa54c08d870 con 0x7fa55c0ff3b0 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.993+0000 7fa55ad9d700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] conn(0x7fa544072350 0x7fa544074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.993+0000 7fa559d9b700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa54406c680 0x7fa54406eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa5500059d0 tx=0x7fa550005960 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.993+0000 7fa55ad9d700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] conn(0x7fa544072350 0x7fa544074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:14.994+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== osd.5 v2:172.21.15.189:6816/580752685 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa544074e10 con 0x7fa544072350 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.010+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 --> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa55c04f2a0 con 0x7fa544072350 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa54b7fe700 1 -- 172.21.15.37:0/69856140 <== osd.5 v2:172.21.15.189:6816/580752685 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa55c04f2a0 con 0x7fa544072350 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] conn(0x7fa544072350 msgr2=0x7fa544074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] conn(0x7fa544072350 0x7fa544074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa54406c680 msgr2=0x7fa54406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa54406c680 0x7fa54406eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa5500059d0 tx=0x7fa550005960 comp rx=0 tx=0).stop 2024-04-15T22:56:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 msgr2=0x7fa55c196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c196df0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00b4b0 tx=0x7fa54c00b870 comp rx=0 tx=0).stop 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 shutdown_connections 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:6816/580752685,v1:172.21.15.189:6817/580752685] conn(0x7fa544072350 0x7fa544074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa54406c680 0x7fa54406eb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa55c0ff3b0 0x7fa55c196df0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 --2- 172.21.15.37:0/69856140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55c0ffd50 0x7fa55c197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 >> 172.21.15.37:0/69856140 conn(0x7fa55c0fac90 msgr2=0x7fa55c0fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 shutdown_connections 2024-04-15T22:56:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:15.011+0000 7fa560ca4700 1 -- 172.21.15.37:0/69856140 wait complete. 2024-04-15T22:56:17.533 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:17 smithi037 ceph-mon[29191]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 -- 172.21.15.37:0/1745746187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 msgr2=0x7f51ec1094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/1745746187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec1094d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009910 tx=0x7f51d4009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 -- 172.21.15.37:0/1745746187 shutdown_connections 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/1745746187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec1094d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/1745746187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 0x7f51ec100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 -- 172.21.15.37:0/1745746187 >> 172.21.15.37:0/1745746187 conn(0x7f51ec0748b0 msgr2=0x7f51ec074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 -- 172.21.15.37:0/1745746187 shutdown_connections 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 -- 172.21.15.37:0/1745746187 wait complete. 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.551+0000 7f51f1ebb700 1 Processor -- start 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51f1ebb700 1 -- start start 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51f1ebb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 0x7f51ec196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51f1ebb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51f1ebb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec197840 con 0x7f51ec100d70 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51f1ebb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec197980 con 0x7f51ec100460 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec197220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46556/0 (socket says 172.21.15.37:46556) 2024-04-15T22:56:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eaffd700 1 -- 172.21.15.37:0/3891405396 learned_addr learned my addr 172.21.15.37:0/3891405396 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eb7fe700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 0x7f51ec196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eaffd700 1 -- 172.21.15.37:0/3891405396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 msgr2=0x7f51ec196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eaffd700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 0x7f51ec196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eaffd700 1 -- 172.21.15.37:0/3891405396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40095f0 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.552+0000 7f51eb7fe700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 0x7f51ec196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.553+0000 7f51eaffd700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec197220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f51d4006010 tx=0x7f51d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.553+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51d401d070 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.553+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec19fd50 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.553+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f51d4004f50 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.553+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51d4021640 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.553+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec1a0160 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.554+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51ec10c580 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f51d400f460 con 0x7f51ec100d70 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51e8ff9700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f51d806c680 0x7f51d806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f51d408c770 con 0x7f51ec100d70 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51eb7fe700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f51d806c680 0x7f51d806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51e8ff9700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] conn(0x7f51d8072350 0x7f51d8074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 --> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51d8074e10 con 0x7f51d8072350 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.555+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51d408ee40 con 0x7f51ec100d70 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.556+0000 7f51ebfff700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] conn(0x7f51d8072350 0x7f51d8074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.556+0000 7f51eb7fe700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f51d806c680 0x7f51d806eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51dc005950 tx=0x7f51dc0058e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.556+0000 7f51ebfff700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] conn(0x7f51d8072350 0x7f51d8074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.557+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== osd.0 v2:172.21.15.37:6802/2814326386 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f51d8074e10 con 0x7f51d8072350 2024-04-15T22:56:17.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.575+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 --> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51ec04f2a0 con 0x7f51d8072350 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.575+0000 7f51e8ff9700 1 -- 172.21.15.37:0/3891405396 <== osd.0 v2:172.21.15.37:6802/2814326386 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f51ec04f2a0 con 0x7f51d8072350 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.575+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] conn(0x7f51d8072350 msgr2=0x7f51d8074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] conn(0x7f51d8072350 0x7f51d8074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f51d806c680 msgr2=0x7f51d806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f51d806c680 0x7f51d806eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51dc005950 tx=0x7f51dc0058e0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 msgr2=0x7f51ec197220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec197220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f51d4006010 tx=0x7f51d400be20 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 shutdown_connections 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6802/2814326386,v1:172.21.15.37:6803/2814326386] conn(0x7f51d8072350 0x7f51d8074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f51d806c680 0x7f51d806eb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51ec100460 0x7f51ec196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 --2- 172.21.15.37:0/3891405396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51ec100d70 0x7f51ec197220 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 >> 172.21.15.37:0/3891405396 conn(0x7f51ec0748b0 msgr2=0x7f51ec103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 shutdown_connections 2024-04-15T22:56:17.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:17.576+0000 7f51f1ebb700 1 -- 172.21.15.37:0/3891405396 wait complete. 2024-04-15T22:56:17.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:17 smithi189 ceph-mon[36179]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:17.981 INFO:teuthology.orchestra.run.smithi037.stdout:167503724552 2024-04-15T22:56:17.982 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-15T22:56:18.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:18 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:18.753 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:18 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:19.698 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:19 smithi037 ceph-mon[29191]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:19.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:19 smithi189 ceph-mon[36179]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 -- 172.21.15.37:0/4009957494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd81004e0 msgr2=0x7f8cd81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 --2- 172.21.15.37:0/4009957494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd81004e0 0x7f8cd81008b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009910 tx=0x7f8cd4009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 -- 172.21.15.37:0/4009957494 shutdown_connections 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 --2- 172.21.15.37:0/4009957494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd8100e80 0x7f8cd8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 --2- 172.21.15.37:0/4009957494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd81004e0 0x7f8cd81008b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 -- 172.21.15.37:0/4009957494 >> 172.21.15.37:0/4009957494 conn(0x7f8cd80fbda0 msgr2=0x7f8cd80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:20.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 -- 172.21.15.37:0/4009957494 shutdown_connections 2024-04-15T22:56:20.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.456+0000 7f8ce0381700 1 -- 172.21.15.37:0/4009957494 wait complete. 2024-04-15T22:56:20.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8ce0381700 1 Processor -- start 2024-04-15T22:56:20.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8ce0381700 1 -- start start 2024-04-15T22:56:20.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8ce0381700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd81004e0 0x7f8cd8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:20.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8ce0381700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 0x7f8cd8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:20.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8ce0381700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd8073ad0 con 0x7f8cd8100e80 2024-04-15T22:56:20.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8ce0381700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd8073c10 con 0x7f8cd81004e0 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8cdd91c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 0x7f8cd8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8cdd91c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 0x7f8cd8073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33752/0 (socket says 172.21.15.37:33752) 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8cdd91c700 1 -- 172.21.15.37:0/2619579444 learned_addr learned my addr 172.21.15.37:0/2619579444 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.457+0000 7f8cde11d700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd81004e0 0x7f8cd8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8cdd91c700 1 -- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd81004e0 msgr2=0x7f8cd8073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8cdd91c700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd81004e0 0x7f8cd8073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8cdd91c700 1 -- 172.21.15.37:0/2619579444 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40095f0 con 0x7f8cd8100e80 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8cdd91c700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 0x7f8cd8073590 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00d900 tx=0x7f8ccc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ccc009900 con 0x7f8cd8100e80 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd81a20c0 con 0x7f8cd8100e80 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8ccc010460 con 0x7f8cd8100e80 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ccc00f5f0 con 0x7f8cd8100e80 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.458+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd81a24d0 con 0x7f8cd8100e80 2024-04-15T22:56:20.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.459+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cd810d670 con 0x7f8cd8100e80 2024-04-15T22:56:20.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.459+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f8ccc0105d0 con 0x7f8cd8100e80 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8ccb7fe700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8cc406c680 0x7f8cc406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8ccc08b570 con 0x7f8cd8100e80 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8cde11d700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8cc406c680 0x7f8cc406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8ccb7fe700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] conn(0x7f8cc4072350 0x7f8cc4074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 --> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cc4074e10 con 0x7f8cc4072350 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ccc08b930 con 0x7f8cd8100e80 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8cde91e700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] conn(0x7f8cc4072350 0x7f8cc4074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.460+0000 7f8cde11d700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8cc406c680 0x7f8cc406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400b5c0 tx=0x7f8cd401a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:20.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.461+0000 7f8cde91e700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] conn(0x7f8cc4072350 0x7f8cc4074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:20.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.463+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== osd.3 v2:172.21.15.189:6800/1840940998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8cc4074e10 con 0x7f8cc4072350 2024-04-15T22:56:20.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.479+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 --> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8cd804f2a0 con 0x7f8cc4072350 2024-04-15T22:56:20.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.479+0000 7f8ccb7fe700 1 -- 172.21.15.37:0/2619579444 <== osd.3 v2:172.21.15.189:6800/1840940998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8cd804f2a0 con 0x7f8cc4072350 2024-04-15T22:56:20.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.479+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] conn(0x7f8cc4072350 msgr2=0x7f8cc4074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:20.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.479+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] conn(0x7f8cc4072350 0x7f8cc4074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8cc406c680 msgr2=0x7f8cc406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8cc406c680 0x7f8cc406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400b5c0 tx=0x7f8cd401a040 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 msgr2=0x7f8cd8073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 0x7f8cd8073590 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00d900 tx=0x7f8ccc00dc10 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 shutdown_connections 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:6800/1840940998,v1:172.21.15.189:6801/1840940998] conn(0x7f8cc4072350 0x7f8cc4074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f8cc406c680 0x7f8cc406eb30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd81004e0 0x7f8cd8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 --2- 172.21.15.37:0/2619579444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd8100e80 0x7f8cd8073590 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 >> 172.21.15.37:0/2619579444 conn(0x7f8cd80fbda0 msgr2=0x7f8cd80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 shutdown_connections 2024-04-15T22:56:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:20.480+0000 7f8ce0381700 1 -- 172.21.15.37:0/2619579444 wait complete. 2024-04-15T22:56:21.613 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:21 smithi037 ceph-mon[29191]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:21.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:21 smithi189 ceph-mon[36179]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:22.024 INFO:teuthology.orchestra.run.smithi037.stdout:38654705699 2024-04-15T22:56:22.024 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-15T22:56:22.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 -- 172.21.15.37:0/1295081116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18069140 msgr2=0x7f3b18104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:22.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 --2- 172.21.15.37:0/1295081116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18069140 0x7f3b18104eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009910 tx=0x7f3b0c009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 -- 172.21.15.37:0/1295081116 shutdown_connections 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 --2- 172.21.15.37:0/1295081116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b18069140 0x7f3b18104eb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 --2- 172.21.15.37:0/1295081116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b180687a0 0x7f3b18068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 -- 172.21.15.37:0/1295081116 >> 172.21.15.37:0/1295081116 conn(0x7f3b18075130 msgr2=0x7f3b18075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 -- 172.21.15.37:0/1295081116 shutdown_connections 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.927+0000 7f3b20560700 1 -- 172.21.15.37:0/1295081116 wait complete. 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b20560700 1 Processor -- start 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b20560700 1 -- start start 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b20560700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 0x7f3b180ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b20560700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b18069140 0x7f3b181001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b20560700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b18100720 con 0x7f3b180687a0 2024-04-15T22:56:22.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b20560700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b18100860 con 0x7f3b18069140 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 0x7f3b180ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 0x7f3b180ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33766/0 (socket says 172.21.15.37:33766) 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 -- 172.21.15.37:0/361374000 learned_addr learned my addr 172.21.15.37:0/361374000 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1dafb700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b18069140 0x7f3b181001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 -- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b18069140 msgr2=0x7f3b181001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b18069140 0x7f3b181001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 -- 172.21.15.37:0/361374000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c0095f0 con 0x7f3b180687a0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.928+0000 7f3b1e2fc700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 0x7f3b180ffca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400d8d0 tx=0x7f3b1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.929+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b14009940 con 0x7f3b180687a0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.929+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b14010460 con 0x7f3b180687a0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.929+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b181a2030 con 0x7f3b180687a0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.929+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b1400f5f0 con 0x7f3b180687a0 2024-04-15T22:56:22.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.929+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b181a2440 con 0x7f3b180687a0 2024-04-15T22:56:22.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.930+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b1810d650 con 0x7f3b180687a0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.930+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3b140105d0 con 0x7f3b180687a0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b0b7fe700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0406c680 0x7f3b0406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3b1408ba40 con 0x7f3b180687a0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b1dafb700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0406c680 0x7f3b0406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b0b7fe700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] conn(0x7f3b04072350 0x7f3b04074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 --> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b04074e10 con 0x7f3b04072350 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b1408bea0 con 0x7f3b180687a0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b1eafd700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] conn(0x7f3b04072350 0x7f3b04074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.931+0000 7f3b1dafb700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0406c680 0x7f3b0406eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3b18101490 tx=0x7f3b0c01a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:22.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.932+0000 7f3b1eafd700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] conn(0x7f3b04072350 0x7f3b04074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:22.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.933+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== osd.4 v2:172.21.15.189:6808/2518250570 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3b04074e10 con 0x7f3b04072350 2024-04-15T22:56:22.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.949+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 --> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3b1804f2a0 con 0x7f3b04072350 2024-04-15T22:56:22.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.949+0000 7f3b0b7fe700 1 -- 172.21.15.37:0/361374000 <== osd.4 v2:172.21.15.189:6808/2518250570 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3b1804f2a0 con 0x7f3b04072350 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.949+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] conn(0x7f3b04072350 msgr2=0x7f3b04074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.949+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] conn(0x7f3b04072350 0x7f3b04074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.949+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0406c680 msgr2=0x7f3b0406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.949+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0406c680 0x7f3b0406eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3b18101490 tx=0x7f3b0c01a040 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 msgr2=0x7f3b180ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 0x7f3b180ffca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400d8d0 tx=0x7f3b1400dc90 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 shutdown_connections 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:6808/2518250570,v1:172.21.15.189:6809/2518250570] conn(0x7f3b04072350 0x7f3b04074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f3b0406c680 0x7f3b0406eb30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b180687a0 0x7f3b180ffca0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 --2- 172.21.15.37:0/361374000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b18069140 0x7f3b181001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 >> 172.21.15.37:0/361374000 conn(0x7f3b18075130 msgr2=0x7f3b18104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 shutdown_connections 2024-04-15T22:56:22.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:22.950+0000 7f3b20560700 1 -- 172.21.15.37:0/361374000 wait complete. 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 -- 172.21.15.37:0/3548832166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ffd60 msgr2=0x7fbedc103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 --2- 172.21.15.37:0/3548832166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ffd60 0x7fbedc103d30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009910 tx=0x7fbed0009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 -- 172.21.15.37:0/3548832166 shutdown_connections 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 --2- 172.21.15.37:0/3548832166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ffd60 0x7fbedc103d30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 --2- 172.21.15.37:0/3548832166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 -- 172.21.15.37:0/3548832166 >> 172.21.15.37:0/3548832166 conn(0x7fbedc0fac60 msgr2=0x7fbedc0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 -- 172.21.15.37:0/3548832166 shutdown_connections 2024-04-15T22:56:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.180+0000 7fbee3719700 1 -- 172.21.15.37:0/3548832166 wait complete. 2024-04-15T22:56:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee3719700 1 Processor -- start 2024-04-15T22:56:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee3719700 1 -- start start 2024-04-15T22:56:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee3719700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee3719700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbedc0ffd60 0x7fbedc197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee3719700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc197860 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee3719700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1979d0 con 0x7fbedc0ffd60 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee14b5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee14b5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33796/0 (socket says 172.21.15.37:33796) 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.181+0000 7fbee14b5700 1 -- 172.21.15.37:0/1118759574 learned_addr learned my addr 172.21.15.37:0/1118759574 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee0cb4700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbedc0ffd60 0x7fbedc197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee14b5700 1 -- 172.21.15.37:0/1118759574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbedc0ffd60 msgr2=0x7fbedc197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee14b5700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbedc0ffd60 0x7fbedc197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee14b5700 1 -- 172.21.15.37:0/1118759574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed00095f0 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee14b5700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc196d00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d8d0 tx=0x7fbed800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbed8009940 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbed8010460 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc19fd70 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbed800f5f0 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.182+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc1a0230 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.183+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbedc10c4f0 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fbed80105d0 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbece7fc700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fbec806c680 0x7fbec806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbed808ba50 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbee0cb4700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fbec806c680 0x7fbec806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbece7fc700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] conn(0x7fbec8072350 0x7fbec8074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 --> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbec8074e10 con 0x7fbec8072350 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.184+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbed808beb0 con 0x7fbedc0ff3c0 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.185+0000 7fbee1cb6700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] conn(0x7fbec8072350 0x7fbec8074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.185+0000 7fbee0cb4700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fbec806c680 0x7fbec806eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbed0006010 tx=0x7fbed001a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.185+0000 7fbee1cb6700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] conn(0x7fbec8072350 0x7fbec8074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.186+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== osd.1 v2:172.21.15.37:6810/1887622200 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbec8074e10 con 0x7fbec8072350 2024-04-15T22:56:23.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.220+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 --> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbedc067980 con 0x7fbec8072350 2024-04-15T22:56:23.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.220+0000 7fbece7fc700 1 -- 172.21.15.37:0/1118759574 <== osd.1 v2:172.21.15.37:6810/1887622200 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbedc067980 con 0x7fbec8072350 2024-04-15T22:56:23.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] conn(0x7fbec8072350 msgr2=0x7fbec8074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] conn(0x7fbec8072350 0x7fbec8074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fbec806c680 msgr2=0x7fbec806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fbec806c680 0x7fbec806eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbed0006010 tx=0x7fbed001a040 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 msgr2=0x7fbedc196d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc196d00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d8d0 tx=0x7fbed800dc90 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 shutdown_connections 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6810/1887622200,v1:172.21.15.37:6811/1887622200] conn(0x7fbec8072350 0x7fbec8074760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fbec806c680 0x7fbec806eb30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbedc0ff3c0 0x7fbedc196d00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 --2- 172.21.15.37:0/1118759574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbedc0ffd60 0x7fbedc197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 >> 172.21.15.37:0/1118759574 conn(0x7fbedc0fac60 msgr2=0x7fbedc0fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 shutdown_connections 2024-04-15T22:56:23.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:23.221+0000 7fbee3719700 1 -- 172.21.15.37:0/1118759574 wait complete. 2024-04-15T22:56:23.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:23 smithi037 ceph-mon[29191]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:23.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:23 smithi189 ceph-mon[36179]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:60129542175 2024-04-15T22:56:24.017 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-15T22:56:24.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 -- 172.21.15.37:0/404867693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80ffbd0 msgr2=0x7fb3c8100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:24.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/404867693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8100040 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009910 tx=0x7fb3b8009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:24.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 -- 172.21.15.37:0/404867693 shutdown_connections 2024-04-15T22:56:24.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/404867693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8100040 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/404867693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c8104130 0x7fb3c8104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 -- 172.21.15.37:0/404867693 >> 172.21.15.37:0/404867693 conn(0x7fb3c8075130 msgr2=0x7fb3c8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:24.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 -- 172.21.15.37:0/404867693 shutdown_connections 2024-04-15T22:56:24.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.215+0000 7fb3cff8f700 1 -- 172.21.15.37:0/404867693 wait complete. 2024-04-15T22:56:24.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cff8f700 1 Processor -- start 2024-04-15T22:56:24.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cff8f700 1 -- start start 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cff8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cff8f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 0x7fb3c8195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cff8f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8195950 con 0x7fb3c8104130 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cff8f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c81996e0 con 0x7fb3c80ffbd0 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cd52a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 0x7fb3c8195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cd52a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 0x7fb3c8195270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33810/0 (socket says 172.21.15.37:33810) 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cd52a700 1 -- 172.21.15.37:0/1798728413 learned_addr learned my addr 172.21.15.37:0/1798728413 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.216+0000 7fb3cdd2b700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cd52a700 1 -- 172.21.15.37:0/1798728413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c80ffbd0 msgr2=0x7fb3c8194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cd52a700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cd52a700 1 -- 172.21.15.37:0/1798728413 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b80095f0 con 0x7fb3c8104130 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cdd2b700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cd52a700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 0x7fb3c8195270 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3b800bd70 tx=0x7fb3b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:24.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3b801d070 con 0x7fb3c8104130 2024-04-15T22:56:24.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb3b800f460 con 0x7fb3c8104130 2024-04-15T22:56:24.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c8199960 con 0x7fb3c8104130 2024-04-15T22:56:24.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3b8017630 con 0x7fb3c8104130 2024-04-15T22:56:24.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.217+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c8199e50 con 0x7fb3c8104130 2024-04-15T22:56:24.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.218+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3c810c5c0 con 0x7fb3c8104130 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb3b8003a60 con 0x7fb3c8104130 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3beffd700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb3b406c680 0x7fb3b406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb3b808cd70 con 0x7fb3c8104130 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3cdd2b700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb3b406c680 0x7fb3b406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3beffd700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] conn(0x7fb3b4072350 0x7fb3b4074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 --> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3b4074e10 con 0x7fb3b4072350 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3b808d1d0 con 0x7fb3c8104130 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.219+0000 7fb3ce52c700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] conn(0x7fb3b4072350 0x7fb3b4074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:24.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.220+0000 7fb3cdd2b700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb3b406c680 0x7fb3b406eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4005950 tx=0x7fb3c40058e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:24.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.220+0000 7fb3ce52c700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] conn(0x7fb3b4072350 0x7fb3b4074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:24.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.220+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== osd.2 v2:172.21.15.37:6818/503233976 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb3b4074e10 con 0x7fb3b4072350 2024-04-15T22:56:24.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 --> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb3c804f2a0 con 0x7fb3b4072350 2024-04-15T22:56:24.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3beffd700 1 -- 172.21.15.37:0/1798728413 <== osd.2 v2:172.21.15.37:6818/503233976 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb3c804f2a0 con 0x7fb3b4072350 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] conn(0x7fb3b4072350 msgr2=0x7fb3b4074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] conn(0x7fb3b4072350 0x7fb3b4074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb3b406c680 msgr2=0x7fb3b406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb3b406c680 0x7fb3b406eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4005950 tx=0x7fb3c40058e0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 msgr2=0x7fb3c8195270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 0x7fb3c8195270 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3b800bd70 tx=0x7fb3b800be50 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 shutdown_connections 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb3b406c680 0x7fb3b406eb30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:6818/503233976,v1:172.21.15.37:6819/503233976] conn(0x7fb3b4072350 0x7fb3b4074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c80ffbd0 0x7fb3c8194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 --2- 172.21.15.37:0/1798728413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c8104130 0x7fb3c8195270 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.237+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 >> 172.21.15.37:0/1798728413 conn(0x7fb3c8075130 msgr2=0x7fb3c80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:24.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.238+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 shutdown_connections 2024-04-15T22:56:24.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:24.238+0000 7fb3cff8f700 1 -- 172.21.15.37:0/1798728413 wait complete. 2024-04-15T22:56:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:146028888078 2024-04-15T22:56:24.773 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-15T22:56:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:25.415 INFO:teuthology.orchestra.run.smithi037.stdout:111669149715 2024-04-15T22:56:25.416 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-15T22:56:25.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:25 smithi037 ceph-mon[29191]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:25.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:25 smithi189 ceph-mon[36179]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:25.910 INFO:teuthology.orchestra.run.smithi037.stdout:81604378649 2024-04-15T22:56:25.910 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-15T22:56:26.578 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:26.618 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:26.619 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:27.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:27 smithi189 ceph-mon[36179]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:27.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:27 smithi037 ceph-mon[29191]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:27.994 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:27.994 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:29.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 -- 172.21.15.37:0/2152692419 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ff440 msgr2=0x7f81000ff810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 --2- 172.21.15.37:0/2152692419 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ff440 0x7f81000ff810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009910 tx=0x7f80f0009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 -- 172.21.15.37:0/2152692419 shutdown_connections 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 --2- 172.21.15.37:0/2152692419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81000ffde0 0x7f8100103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 --2- 172.21.15.37:0/2152692419 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ff440 0x7f81000ff810 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 -- 172.21.15.37:0/2152692419 >> 172.21.15.37:0/2152692419 conn(0x7f81000face0 msgr2=0x7f81000fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 -- 172.21.15.37:0/2152692419 shutdown_connections 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.659+0000 7f8105c2d700 1 -- 172.21.15.37:0/2152692419 wait complete. 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.660+0000 7f8105c2d700 1 Processor -- start 2024-04-15T22:56:29.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.660+0000 7f8105c2d700 1 -- start start 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.660+0000 7f8105c2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81000ff440 0x7f8100195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.660+0000 7f8105c2d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 0x7f81001963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.660+0000 7f8105c2d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100196ab0 con 0x7f81000ffde0 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.660+0000 7f8105c2d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810019a840 con 0x7f81000ff440 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 0x7f81001963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 0x7f81001963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46694/0 (socket says 172.21.15.37:46694) 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 -- 172.21.15.37:0/3389769660 learned_addr learned my addr 172.21.15.37:0/3389769660 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80ff7fe700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81000ff440 0x7f8100195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 -- 172.21.15.37:0/3389769660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81000ff440 msgr2=0x7f8100195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81000ff440 0x7f8100195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 -- 172.21.15.37:0/3389769660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f00095f0 con 0x7f81000ffde0 2024-04-15T22:56:29.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80f6ffd700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 0x7f81001963d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f80e800c8f0 tx=0x7f80e800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:29.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80e80078a0 con 0x7f81000ffde0 2024-04-15T22:56:29.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f80e800f450 con 0x7f81000ffde0 2024-04-15T22:56:29.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810019ab20 con 0x7f81000ffde0 2024-04-15T22:56:29.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80e8018610 con 0x7f81000ffde0 2024-04-15T22:56:29.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.661+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810019b070 con 0x7f81000ffde0 2024-04-15T22:56:29.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.662+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8100108760 con 0x7f81000ffde0 2024-04-15T22:56:29.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.663+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f80e800f5c0 con 0x7f81000ffde0 2024-04-15T22:56:29.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.663+0000 7f80fd7fa700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f80ec06c680 0x7f80ec06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:29.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.663+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f80e808b490 con 0x7f81000ffde0 2024-04-15T22:56:29.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.663+0000 7f80ff7fe700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f80ec06c680 0x7f80ec06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:29.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.664+0000 7f80ff7fe700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f80ec06c680 0x7f80ec06eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80f0006010 tx=0x7f80f001a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:29.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.668+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f80e8055eb0 con 0x7f81000ffde0 2024-04-15T22:56:29.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:29 smithi189 ceph-mon[36179]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:29.777 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:29 smithi037 ceph-mon[29191]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:29.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.880+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f810019b2f0 con 0x7f81000ffde0 2024-04-15T22:56:29.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.880+0000 7f80fd7fa700 1 -- 172.21.15.37:0/3389769660 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f80e801f300 con 0x7f81000ffde0 2024-04-15T22:56:29.873 INFO:teuthology.orchestra.run.smithi037.stdout:81604378650 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f80ec06c680 msgr2=0x7f80ec06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f80ec06c680 0x7f80ec06eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80f0006010 tx=0x7f80f001a040 comp rx=0 tx=0).stop 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 msgr2=0x7f81001963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 0x7f81001963d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f80e800c8f0 tx=0x7f80e800cc00 comp rx=0 tx=0).stop 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 shutdown_connections 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f80ec06c680 0x7f80ec06eb30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81000ff440 0x7f8100195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 --2- 172.21.15.37:0/3389769660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81000ffde0 0x7f81001963d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 >> 172.21.15.37:0/3389769660 conn(0x7f81000face0 msgr2=0x7f8100069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 shutdown_connections 2024-04-15T22:56:29.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:29.882+0000 7f8105c2d700 1 -- 172.21.15.37:0/3389769660 wait complete. 2024-04-15T22:56:30.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:30 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3389769660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:56:30.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:30 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3389769660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:56:31.558 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378650 for osd.2 2024-04-15T22:56:31.559 DEBUG:teuthology.parallel:result is None 2024-04-15T22:56:31.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:31 smithi189 ceph-mon[36179]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:31.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:31 smithi037 ceph-mon[29191]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:32.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 -- 172.21.15.37:0/853043064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 msgr2=0x7f2680100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 --2- 172.21.15.37:0/853043064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680100040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2668009910 tx=0x7f2668009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 -- 172.21.15.37:0/853043064 shutdown_connections 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 --2- 172.21.15.37:0/853043064 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680100040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 --2- 172.21.15.37:0/853043064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2680104130 0x7f2680104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 -- 172.21.15.37:0/853043064 >> 172.21.15.37:0/853043064 conn(0x7f2680075130 msgr2=0x7f2680075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 -- 172.21.15.37:0/853043064 shutdown_connections 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.039+0000 7f26853c5700 1 -- 172.21.15.37:0/853043064 wait complete. 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f26853c5700 1 Processor -- start 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f26853c5700 1 -- start start 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f26853c5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:32.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f26853c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2680104130 0x7f2680197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f26853c5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680197880 con 0x7f26800ffbd0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f26853c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26801979c0 con 0x7f2680104130 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f267effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f267effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46700/0 (socket says 172.21.15.37:46700) 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f267effd700 1 -- 172.21.15.37:0/3048375480 learned_addr learned my addr 172.21.15.37:0/3048375480 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.040+0000 7f267e7fc700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2680104130 0x7f2680197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f267effd700 1 -- 172.21.15.37:0/3048375480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2680104130 msgr2=0x7f2680197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f267effd700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2680104130 0x7f2680197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f267effd700 1 -- 172.21.15.37:0/3048375480 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680095f0 con 0x7f26800ffbd0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f267effd700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680196d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f267000b4b0 tx=0x7f267000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2670011800 con 0x7f26800ffbd0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2670011e10 con 0x7f26800ffbd0 2024-04-15T22:56:32.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268019fd90 con 0x7f26800ffbd0 2024-04-15T22:56:32.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f267000f2e0 con 0x7f26800ffbd0 2024-04-15T22:56:32.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.041+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26801a0250 con 0x7f26800ffbd0 2024-04-15T22:56:32.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.042+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26801076a0 con 0x7f26800ffbd0 2024-04-15T22:56:32.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.042+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2670010380 con 0x7f26800ffbd0 2024-04-15T22:56:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.043+0000 7f2677fff700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f266c06c680 0x7f266c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.043+0000 7f267e7fc700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f266c06c680 0x7f266c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.043+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f267008b100 con 0x7f26800ffbd0 2024-04-15T22:56:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.043+0000 7f267e7fc700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f266c06c680 0x7f266c06eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2680068df0 tx=0x7f266801a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.047+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2670055b20 con 0x7f26800ffbd0 2024-04-15T22:56:32.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.252+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f268004f2a0 con 0x7f26800ffbd0 2024-04-15T22:56:32.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.253+0000 7f2677fff700 1 -- 172.21.15.37:0/3048375480 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2670059140 con 0x7f26800ffbd0 2024-04-15T22:56:32.245 INFO:teuthology.orchestra.run.smithi037.stdout:167503724555 2024-04-15T22:56:32.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f266c06c680 msgr2=0x7f266c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:32.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f266c06c680 0x7f266c06eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2680068df0 tx=0x7f266801a040 comp rx=0 tx=0).stop 2024-04-15T22:56:32.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 msgr2=0x7f2680196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680196d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f267000b4b0 tx=0x7f267000b7c0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 shutdown_connections 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f266c06c680 0x7f266c06eb30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26800ffbd0 0x7f2680196d20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 --2- 172.21.15.37:0/3048375480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2680104130 0x7f2680197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 >> 172.21.15.37:0/3048375480 conn(0x7f2680075130 msgr2=0x7f26800fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 shutdown_connections 2024-04-15T22:56:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:32.255+0000 7f26853c5700 1 -- 172.21.15.37:0/3048375480 wait complete. 2024-04-15T22:56:32.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:32 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3048375480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:56:32.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:32 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3048375480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:56:33.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:33 smithi189 ceph-mon[36179]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:33.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:33 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:33.751 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:33 smithi037 ceph-mon[29191]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:33.751 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:33 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 -- 172.21.15.37:0/4036465787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 msgr2=0x7f3118100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 --2- 172.21.15.37:0/4036465787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118100040 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f310c009910 tx=0x7f310c009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 -- 172.21.15.37:0/4036465787 shutdown_connections 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 --2- 172.21.15.37:0/4036465787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118100040 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 --2- 172.21.15.37:0/4036465787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3118104130 0x7f3118104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 -- 172.21.15.37:0/4036465787 >> 172.21.15.37:0/4036465787 conn(0x7f3118075130 msgr2=0x7f3118075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 -- 172.21.15.37:0/4036465787 shutdown_connections 2024-04-15T22:56:35.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.214+0000 7f311f393700 1 -- 172.21.15.37:0/4036465787 wait complete. 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311f393700 1 Processor -- start 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311f393700 1 -- start start 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311f393700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311f393700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3118104130 0x7f31181971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311f393700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31181977e0 con 0x7f31180ffbd0 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311f393700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3118197950 con 0x7f3118104130 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.215+0000 7f311d12f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311d12f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46732/0 (socket says 172.21.15.37:46732) 2024-04-15T22:56:35.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311d12f700 1 -- 172.21.15.37:0/3206068698 learned_addr learned my addr 172.21.15.37:0/3206068698 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311d12f700 1 -- 172.21.15.37:0/3206068698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3118104130 msgr2=0x7f31181971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311d12f700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3118104130 0x7f31181971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311d12f700 1 -- 172.21.15.37:0/3206068698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0095f0 con 0x7f31180ffbd0 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311d12f700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118196c80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f311400d8d0 tx=0x7f311400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3114009940 con 0x7f31180ffbd0 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311819fcf0 con 0x7f31180ffbd0 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3114010460 con 0x7f31180ffbd0 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f311400f5f0 con 0x7f31180ffbd0 2024-04-15T22:56:35.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.216+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31181a01e0 con 0x7f31180ffbd0 2024-04-15T22:56:35.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.217+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311804f2a0 con 0x7f31180ffbd0 2024-04-15T22:56:35.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.218+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f311400f830 con 0x7f31180ffbd0 2024-04-15T22:56:35.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.218+0000 7f310a7fc700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f310406c680 0x7f310406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:35.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.218+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f311408b550 con 0x7f31180ffbd0 2024-04-15T22:56:35.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.218+0000 7f311c92e700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f310406c680 0x7f310406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:35.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.219+0000 7f311c92e700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f310406c680 0x7f310406eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f310c009fd0 tx=0x7f310c005fb0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:35.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.223+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3114056050 con 0x7f31180ffbd0 2024-04-15T22:56:35.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.428+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f31181a0510 con 0x7f31180ffbd0 2024-04-15T22:56:35.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.428+0000 7f310a7fc700 1 -- 172.21.15.37:0/3206068698 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3114059670 con 0x7f31180ffbd0 2024-04-15T22:56:35.422 INFO:teuthology.orchestra.run.smithi037.stdout:38654705703 2024-04-15T22:56:35.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f310406c680 msgr2=0x7f310406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f310406c680 0x7f310406eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f310c009fd0 tx=0x7f310c005fb0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 msgr2=0x7f3118196c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118196c80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f311400d8d0 tx=0x7f311400dc90 comp rx=0 tx=0).stop 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 shutdown_connections 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f310406c680 0x7f310406eb30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31180ffbd0 0x7f3118196c80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 --2- 172.21.15.37:0/3206068698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3118104130 0x7f31181971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 >> 172.21.15.37:0/3206068698 conn(0x7f3118075130 msgr2=0x7f31180feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 shutdown_connections 2024-04-15T22:56:35.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.431+0000 7f311f393700 1 -- 172.21.15.37:0/3206068698 wait complete. 2024-04-15T22:56:35.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 -- 172.21.15.37:0/2679141065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 msgr2=0x7f5e3c100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2679141065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c100890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009910 tx=0x7f5e38009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:35.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 -- 172.21.15.37:0/2679141065 shutdown_connections 2024-04-15T22:56:35.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2679141065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e3c100e60 0x7f5e3c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2679141065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c100890 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 -- 172.21.15.37:0/2679141065 >> 172.21.15.37:0/2679141065 conn(0x7f5e3c0fbd80 msgr2=0x7f5e3c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:35.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 -- 172.21.15.37:0/2679141065 shutdown_connections 2024-04-15T22:56:35.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.433+0000 7f5e443bf700 1 -- 172.21.15.37:0/2679141065 wait complete. 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e443bf700 1 Processor -- start 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e443bf700 1 -- start start 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e443bf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e443bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e3c100e60 0x7f5e3c1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e443bf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c196ab0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e443bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c19a840 con 0x7f5e3c100e60 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46742/0 (socket says 172.21.15.37:46742) 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 -- 172.21.15.37:0/2392423098 learned_addr learned my addr 172.21.15.37:0/2392423098 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4195a700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e3c100e60 0x7f5e3c1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 -- 172.21.15.37:0/2392423098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e3c100e60 msgr2=0x7f5e3c1963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e3c100e60 0x7f5e3c1963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 -- 172.21.15.37:0/2392423098 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380095f0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.434+0000 7f5e4215b700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c195e90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e380092e0 tx=0x7f5e3800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:35.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.435+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e3801d070 con 0x7f5e3c1004c0 2024-04-15T22:56:35.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.435+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c19aac0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.435+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5e38005e00 con 0x7f5e3c1004c0 2024-04-15T22:56:35.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.435+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e38017400 con 0x7f5e3c1004c0 2024-04-15T22:56:35.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.435+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c19afb0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.436+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c1087d0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.436+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f5e380176c0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.437+0000 7f5e337fe700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f5e2806c680 0x7f5e2806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:35.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.437+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f5e3808ca30 con 0x7f5e3c1004c0 2024-04-15T22:56:35.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.437+0000 7f5e4195a700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f5e2806c680 0x7f5e2806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:35.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.438+0000 7f5e4195a700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f5e2806c680 0x7f5e2806eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c1974b0 tx=0x7f5e2c00b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:35.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.442+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5e38057450 con 0x7f5e3c1004c0 2024-04-15T22:56:35.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:35 smithi037 ceph-mon[29191]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:35.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:35 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3936808640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:56:35.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:35 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3206068698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:56:35.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.646+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5e3c04f2a0 con 0x7f5e3c1004c0 2024-04-15T22:56:35.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.647+0000 7f5e337fe700 1 -- 172.21.15.37:0/2392423098 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5e3805aa70 con 0x7f5e3c1004c0 2024-04-15T22:56:35.639 INFO:teuthology.orchestra.run.smithi037.stdout:111669149717 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f5e2806c680 msgr2=0x7f5e2806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f5e2806c680 0x7f5e2806eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c1974b0 tx=0x7f5e2c00b410 comp rx=0 tx=0).stop 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 msgr2=0x7f5e3c195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c195e90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e380092e0 tx=0x7f5e3800bd70 comp rx=0 tx=0).stop 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 shutdown_connections 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f5e2806c680 0x7f5e2806eb30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e3c1004c0 0x7f5e3c195e90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.648+0000 7f5e443bf700 1 --2- 172.21.15.37:0/2392423098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e3c100e60 0x7f5e3c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:35.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.649+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 >> 172.21.15.37:0/2392423098 conn(0x7f5e3c0fbd80 msgr2=0x7f5e3c0fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:35.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.649+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 shutdown_connections 2024-04-15T22:56:35.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.649+0000 7f5e443bf700 1 -- 172.21.15.37:0/2392423098 wait complete. 2024-04-15T22:56:35.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:35 smithi189 ceph-mon[36179]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:35.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:35 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3936808640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:56:35.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:35 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3206068698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:56:36.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149717 for osd.3 2024-04-15T22:56:36.280 DEBUG:teuthology.parallel:result is None 2024-04-15T22:56:36.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:36 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2392423098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:56:36.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724552 got 167503724555 for osd.5 2024-04-15T22:56:36.693 DEBUG:teuthology.parallel:result is None 2024-04-15T22:56:36.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:36 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2392423098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:56:37.231 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705703 for osd.0 2024-04-15T22:56:37.232 DEBUG:teuthology.parallel:result is None 2024-04-15T22:56:37.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.123+0000 7ff13902a700 1 -- 172.21.15.37:0/3637288663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 msgr2=0x7ff1340ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.123+0000 7ff13902a700 1 --2- 172.21.15.37:0/3637288663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff1340ff790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff124009910 tx=0x7ff124009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.123+0000 7ff13902a700 1 -- 172.21.15.37:0/3637288663 shutdown_connections 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.123+0000 7ff13902a700 1 --2- 172.21.15.37:0/3637288663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1340ffd60 0x7ff134103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.123+0000 7ff13902a700 1 --2- 172.21.15.37:0/3637288663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff1340ff790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.123+0000 7ff13902a700 1 -- 172.21.15.37:0/3637288663 >> 172.21.15.37:0/3637288663 conn(0x7ff1340fac60 msgr2=0x7ff1340fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.124+0000 7ff13902a700 1 -- 172.21.15.37:0/3637288663 shutdown_connections 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.124+0000 7ff13902a700 1 -- 172.21.15.37:0/3637288663 wait complete. 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.124+0000 7ff13902a700 1 Processor -- start 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.124+0000 7ff13902a700 1 -- start start 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff13902a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff134195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff13902a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1340ffd60 0x7ff134196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff13902a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff134196b50 con 0x7ff1340ff3c0 2024-04-15T22:56:37.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff13902a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13419a8e0 con 0x7ff1340ffd60 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff134195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff134195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46720/0 (socket says 172.21.15.37:46720) 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 -- 172.21.15.37:0/3936808640 learned_addr learned my addr 172.21.15.37:0/3936808640 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff13259c700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1340ffd60 0x7ff134196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 -- 172.21.15.37:0/3936808640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1340ffd60 msgr2=0x7ff134196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1340ffd60 0x7ff134196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 -- 172.21.15.37:0/3936808640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240095f0 con 0x7ff1340ff3c0 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff132d9d700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff134195f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff124006010 tx=0x7ff12400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff12401d070 con 0x7ff1340ff3c0 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.125+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13419ab60 con 0x7ff1340ff3c0 2024-04-15T22:56:37.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.126+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff124004f50 con 0x7ff1340ff3c0 2024-04-15T22:56:37.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.126+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13419b050 con 0x7ff1340ff3c0 2024-04-15T22:56:37.326 INFO:teuthology.orchestra.run.smithi037.stdout:146028888080 2024-04-15T22:56:37.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.126+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff124021640 con 0x7ff1340ff3c0 2024-04-15T22:56:37.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.127+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1341086e0 con 0x7ff1340ff3c0 2024-04-15T22:56:37.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.127+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff124004aa0 con 0x7ff1340ff3c0 2024-04-15T22:56:37.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.128+0000 7ff11bfff700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff11c06c680 0x7ff11c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.128+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff12408c790 con 0x7ff1340ff3c0 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.128+0000 7ff13259c700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff11c06c680 0x7ff11c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.128+0000 7ff13259c700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff11c06c680 0x7ff11c06eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff128005950 tx=0x7ff1280058e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.132+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff1240571b0 con 0x7ff1340ff3c0 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.342+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff134197290 con 0x7ff1340ff3c0 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.342+0000 7ff11bfff700 1 -- 172.21.15.37:0/3936808640 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff124026090 con 0x7ff1340ff3c0 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff11c06c680 msgr2=0x7ff11c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff11c06c680 0x7ff11c06eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff128005950 tx=0x7ff1280058e0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 msgr2=0x7ff134195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff134195f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff124006010 tx=0x7ff12400be20 comp rx=0 tx=0).stop 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 shutdown_connections 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff11c06c680 0x7ff11c06eb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1340ff3c0 0x7ff134195f30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 --2- 172.21.15.37:0/3936808640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1340ffd60 0x7ff134196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:37.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.344+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 >> 172.21.15.37:0/3936808640 conn(0x7ff1340fac60 msgr2=0x7ff1340690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:37.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.345+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 shutdown_connections 2024-04-15T22:56:37.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:35.345+0000 7ff13902a700 1 -- 172.21.15.37:0/3936808640 wait complete. 2024-04-15T22:56:37.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:37 smithi037 ceph-mon[29191]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:37.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:37 smithi189 ceph-mon[36179]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:37.970 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888080 for osd.4 2024-04-15T22:56:37.970 DEBUG:teuthology.parallel:result is None 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 -- 172.21.15.37:0/2929239468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ff3c0 msgr2=0x7eff800ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 --2- 172.21.15.37:0/2929239468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ff3c0 0x7eff800ff790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7eff70009910 tx=0x7eff70009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 -- 172.21.15.37:0/2929239468 shutdown_connections 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 --2- 172.21.15.37:0/2929239468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ffd60 0x7eff80103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 --2- 172.21.15.37:0/2929239468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ff3c0 0x7eff800ff790 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 -- 172.21.15.37:0/2929239468 >> 172.21.15.37:0/2929239468 conn(0x7eff800fac80 msgr2=0x7eff800fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 -- 172.21.15.37:0/2929239468 shutdown_connections 2024-04-15T22:56:38.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.309+0000 7eff85d41700 1 -- 172.21.15.37:0/2929239468 wait complete. 2024-04-15T22:56:38.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff85d41700 1 Processor -- start 2024-04-15T22:56:38.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff85d41700 1 -- start start 2024-04-15T22:56:38.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff85d41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ff3c0 0x7eff80195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff85d41700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 0x7eff80196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff85d41700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff80196b20 con 0x7eff800ffd60 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff85d41700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8019a8b0 con 0x7eff800ff3c0 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff7effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 0x7eff80196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff7effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 0x7eff80196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46758/0 (socket says 172.21.15.37:46758) 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.310+0000 7eff7effd700 1 -- 172.21.15.37:0/3914678796 learned_addr learned my addr 172.21.15.37:0/3914678796 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7f7fe700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ff3c0 0x7eff80195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7effd700 1 -- 172.21.15.37:0/3914678796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ff3c0 msgr2=0x7eff80195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7effd700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ff3c0 0x7eff80195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7effd700 1 -- 172.21.15.37:0/3914678796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700095f0 con 0x7eff800ffd60 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7f7fe700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ff3c0 0x7eff80195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T22:56:38.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7effd700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 0x7eff80196440 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff7400d900 tx=0x7eff7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:38.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff74009900 con 0x7eff800ffd60 2024-04-15T22:56:38.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8019ab90 con 0x7eff800ffd60 2024-04-15T22:56:38.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff74010460 con 0x7eff800ffd60 2024-04-15T22:56:38.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff7400f5f0 con 0x7eff800ffd60 2024-04-15T22:56:38.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.311+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8019b0e0 con 0x7eff800ffd60 2024-04-15T22:56:38.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.312+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8004f2a0 con 0x7eff800ffd60 2024-04-15T22:56:38.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.313+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7eff740105d0 con 0x7eff800ffd60 2024-04-15T22:56:38.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.313+0000 7eff7cff9700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7eff6806c680 0x7eff6806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:38.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.313+0000 7eff7f7fe700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7eff6806c680 0x7eff6806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:38.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.313+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7eff7408b1a0 con 0x7eff800ffd60 2024-04-15T22:56:38.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.314+0000 7eff7f7fe700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7eff6806c680 0x7eff6806eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7eff7000b5c0 tx=0x7eff7001a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:38.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.318+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7eff74055bc0 con 0x7eff800ffd60 2024-04-15T22:56:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.520+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7eff8019b3c0 con 0x7eff800ffd60 2024-04-15T22:56:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.520+0000 7eff7cff9700 1 -- 172.21.15.37:0/3914678796 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff740591e0 con 0x7eff800ffd60 2024-04-15T22:56:38.513 INFO:teuthology.orchestra.run.smithi037.stdout:60129542178 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7eff6806c680 msgr2=0x7eff6806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7eff6806c680 0x7eff6806eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7eff7000b5c0 tx=0x7eff7001a040 comp rx=0 tx=0).stop 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 msgr2=0x7eff80196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 0x7eff80196440 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff7400d900 tx=0x7eff7400dcc0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 shutdown_connections 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7eff6806c680 0x7eff6806eb30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff800ff3c0 0x7eff80195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 --2- 172.21.15.37:0/3914678796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff800ffd60 0x7eff80196440 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 >> 172.21.15.37:0/3914678796 conn(0x7eff800fac80 msgr2=0x7eff801034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 shutdown_connections 2024-04-15T22:56:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:38.523+0000 7eff85d41700 1 -- 172.21.15.37:0/3914678796 wait complete. 2024-04-15T22:56:39.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542178 for osd.1 2024-04-15T22:56:39.149 DEBUG:teuthology.parallel:result is None 2024-04-15T22:56:39.149 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T22:56:39.149 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T22:56:39.367 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:39.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:39 smithi037 ceph-mon[29191]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:39.711 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:39 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3914678796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:56:39.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:39 smithi189 ceph-mon[36179]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:39.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:39 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3914678796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:56:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.964+0000 7fe300cd1700 1 -- 172.21.15.37:0/2617134976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 msgr2=0x7fe2fc1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.964+0000 7fe300cd1700 1 --2- 172.21.15.37:0/2617134976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc1008b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009910 tx=0x7fe2ec009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 -- 172.21.15.37:0/2617134976 shutdown_connections 2024-04-15T22:56:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 --2- 172.21.15.37:0/2617134976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2fc100e80 0x7fe2fc104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 --2- 172.21.15.37:0/2617134976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc1008b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 -- 172.21.15.37:0/2617134976 >> 172.21.15.37:0/2617134976 conn(0x7fe2fc0fbda0 msgr2=0x7fe2fc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 -- 172.21.15.37:0/2617134976 shutdown_connections 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 -- 172.21.15.37:0/2617134976 wait complete. 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 Processor -- start 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.965+0000 7fe300cd1700 1 -- start start 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe300cd1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe300cd1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2fc100e80 0x7fe2fc073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe300cd1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc073ad0 con 0x7fe2fc1004e0 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe300cd1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc073c10 con 0x7fe2fc100e80 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32908/0 (socket says 172.21.15.37:32908) 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 -- 172.21.15.37:0/3176414582 learned_addr learned my addr 172.21.15.37:0/3176414582 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2f9d9b700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2fc100e80 0x7fe2fc073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 -- 172.21.15.37:0/3176414582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2fc100e80 msgr2=0x7fe2fc073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2fc100e80 0x7fe2fc073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 -- 172.21.15.37:0/3176414582 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0095f0 con 0x7fe2fc1004e0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2fa59c700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc073050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec006010 tx=0x7fe2ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2ec01d070 con 0x7fe2fc1004e0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe2ec00f460 con 0x7fe2fc1004e0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc073e90 con 0x7fe2fc1004e0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2ec021640 con 0x7fe2fc1004e0 2024-04-15T22:56:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.966+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc1a23f0 con 0x7fe2fc1004e0 2024-04-15T22:56:40.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.967+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc1087f0 con 0x7fe2fc1004e0 2024-04-15T22:56:40.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.968+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe2ec00faf0 con 0x7fe2fc1004e0 2024-04-15T22:56:40.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.968+0000 7fe2f37fe700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe2e806c680 0x7fe2e806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:40.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.969+0000 7fe2f9d9b700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe2e806c680 0x7fe2e806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:40.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.969+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe2ec08c840 con 0x7fe2fc1004e0 2024-04-15T22:56:40.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.969+0000 7fe2f9d9b700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe2e806c680 0x7fe2e806eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4005fd0 tx=0x7fe2e400a560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:40.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:40.973+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe2ec057260 con 0x7fe2fc1004e0 2024-04-15T22:56:41.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.168+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2fc061190 con 0x7fe2e806c680 2024-04-15T22:56:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.170+0000 7fe2f37fe700 1 -- 172.21.15.37:0/3176414582 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28142 (secure 0 0 0) 0x7fe2fc061190 con 0x7fe2e806c680 2024-04-15T22:56:41.163 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:56:41.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe2e806c680 msgr2=0x7fe2e806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:41.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe2e806c680 0x7fe2e806eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4005fd0 tx=0x7fe2e400a560 comp rx=0 tx=0).stop 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 msgr2=0x7fe2fc073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc073050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec006010 tx=0x7fe2ec00be20 comp rx=0 tx=0).stop 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 shutdown_connections 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe2e806c680 0x7fe2e806eb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc1004e0 0x7fe2fc073050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 --2- 172.21.15.37:0/3176414582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2fc100e80 0x7fe2fc073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 >> 172.21.15.37:0/3176414582 conn(0x7fe2fc0fbda0 msgr2=0x7fe2fc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 shutdown_connections 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:41.172+0000 7fe300cd1700 1 -- 172.21.15.37:0/3176414582 wait complete. 2024-04-15T22:56:41.165 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-15T22:56:41.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:41 smithi189 ceph-mon[36179]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:41.771 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:41 smithi037 ceph-mon[29191]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:41.792 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-04-15T22:56:40.384910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165584,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305328,"statfs":{"total":575970213888,"available":575800655872,"internally_reserved":0,"allocated":3964928,"data_stored":2608497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002204"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T22:55:51.867939+0000","last_change":"2024-04-15T22:55:25.358923+0000","last_active":"2024-04-15T22:55:51.867939+0000","last_peered":"2024-04-15T22:55:51.867939+0000","last_clean":"2024-04-15T22:55:51.867939+0000","last_became_active":"2024-04-15T22:55:25.358639+0000","last_became_peered":"2024-04-15T22:55:25.358639+0000","last_unstale":"2024-04-15T22:55:51.867939+0000","last_undegraded":"2024-04-15T22:55:51.867939+0000","last_fullsized":"2024-04-15T22:55:51.867939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T22:54:34.664844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T22:54:34.664844+0000","last_clean_scrub_stamp":"2024-04-15T22:54:34.664844+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T08:47:11.035710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]},{"osd":4,"up_from":34,"seq":146028888081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27304,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717848,"statfs":{"total":95995035648,"available":95967076352,"internally_reserved":0,"allocated":360448,"data_stored":139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 15 22:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61199999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.622},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 22:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.309,"15min":0.29799999999999999},"min":{"1min":0.247,"5min":0.193,"15min":0.193},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.25800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.315}]},{"osd":2,"last update":"Mon Apr 15 22:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.247},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27000000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:56:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32200000000000001},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.316},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.442,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Apr 15 22:56:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Apr 15 22:56:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.35299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T22:56:41.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T22:56:42.010 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:42.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:42 smithi189 ceph-mon[36179]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:56:42.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:42 smithi037 ceph-mon[29191]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:56:43.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 -- 172.21.15.37:0/281666275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 msgr2=0x7ff0fc0ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:43.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 --2- 172.21.15.37:0/281666275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc0ff720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009910 tx=0x7ff0ec009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:43.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 -- 172.21.15.37:0/281666275 shutdown_connections 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 --2- 172.21.15.37:0/281666275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 0x7ff0fc10a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 --2- 172.21.15.37:0/281666275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc0ff720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 -- 172.21.15.37:0/281666275 >> 172.21.15.37:0/281666275 conn(0x7ff0fc0748b0 msgr2=0x7ff0fc074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 -- 172.21.15.37:0/281666275 shutdown_connections 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.619+0000 7ff101d2b700 1 -- 172.21.15.37:0/281666275 wait complete. 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff101d2b700 1 Processor -- start 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff101d2b700 1 -- start start 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff101d2b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff101d2b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 0x7ff0fc100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff101d2b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc100fb0 con 0x7ff0fc0ff350 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff101d2b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc1010f0 con 0x7ff0fc0ffc60 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff0fb7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:43.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff0fb7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc100530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32912/0 (socket says 172.21.15.37:32912) 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff0fb7fe700 1 -- 172.21.15.37:0/3639605161 learned_addr learned my addr 172.21.15.37:0/3639605161 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.620+0000 7ff0faffd700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 0x7ff0fc100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0fb7fe700 1 -- 172.21.15.37:0/3639605161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 msgr2=0x7ff0fc100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0fb7fe700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 0x7ff0fc100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0fb7fe700 1 -- 172.21.15.37:0/3639605161 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec0095f0 con 0x7ff0fc0ff350 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0faffd700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 0x7ff0fc100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0fb7fe700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc100530 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0092e0 tx=0x7ff0ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0ec01d070 con 0x7ff0fc0ff350 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc101370 con 0x7ff0fc0ff350 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0ec00f460 con 0x7ff0fc0ff350 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0ec017630 con 0x7ff0fc0ff350 2024-04-15T22:56:43.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.621+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc1a2300 con 0x7ff0fc0ff350 2024-04-15T22:56:43.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.622+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc107dd0 con 0x7ff0fc0ff350 2024-04-15T22:56:43.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.623+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff0ec00f5d0 con 0x7ff0fc0ff350 2024-04-15T22:56:43.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.623+0000 7ff0f8ff9700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff0e406c680 0x7ff0e406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:43.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.623+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff0ec08c9b0 con 0x7ff0fc0ff350 2024-04-15T22:56:43.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.623+0000 7ff0faffd700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff0e406c680 0x7ff0e406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:43.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.624+0000 7ff0faffd700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff0e406c680 0x7ff0e406eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005950 tx=0x7ff0f00058e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:43.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.628+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff0ec0573d0 con 0x7ff0fc0ff350 2024-04-15T22:56:43.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:43 smithi189 ceph-mon[36179]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:43.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.820+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0fc02cca0 con 0x7ff0e406c680 2024-04-15T22:56:43.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:43 smithi037 ceph-mon[29191]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:43.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.822+0000 7ff0f8ff9700 1 -- 172.21.15.37:0/3639605161 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28112 (secure 0 0 0) 0x7ff0fc02cca0 con 0x7ff0e406c680 2024-04-15T22:56:43.816 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:56:43.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff0e406c680 msgr2=0x7ff0e406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:43.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff0e406c680 0x7ff0e406eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005950 tx=0x7ff0f00058e0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 msgr2=0x7ff0fc100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:43.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc100530 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0092e0 tx=0x7ff0ec00be20 comp rx=0 tx=0).stop 2024-04-15T22:56:43.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 shutdown_connections 2024-04-15T22:56:43.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff0e406c680 0x7ff0e406eb30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0fc0ff350 0x7ff0fc100530 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 --2- 172.21.15.37:0/3639605161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0fc0ffc60 0x7ff0fc100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:43.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 >> 172.21.15.37:0/3639605161 conn(0x7ff0fc0748b0 msgr2=0x7ff0fc104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:43.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 shutdown_connections 2024-04-15T22:56:43.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:43.824+0000 7ff101d2b700 1 -- 172.21.15.37:0/3639605161 wait complete. 2024-04-15T22:56:43.820 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-15T22:56:44.486 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-15T22:56:42.385263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165584,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305328,"statfs":{"total":575970213888,"available":575800655872,"internally_reserved":0,"allocated":3964928,"data_stored":2608497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002151"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T22:55:51.867939+0000","last_change":"2024-04-15T22:55:25.358923+0000","last_active":"2024-04-15T22:55:51.867939+0000","last_peered":"2024-04-15T22:55:51.867939+0000","last_clean":"2024-04-15T22:55:51.867939+0000","last_became_active":"2024-04-15T22:55:25.358639+0000","last_became_peered":"2024-04-15T22:55:25.358639+0000","last_unstale":"2024-04-15T22:55:51.867939+0000","last_undegraded":"2024-04-15T22:55:51.867939+0000","last_fullsized":"2024-04-15T22:55:51.867939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T22:54:34.664844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T22:54:34.664844+0000","last_clean_scrub_stamp":"2024-04-15T22:54:34.664844+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T08:47:11.035710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.433}]},{"osd":1,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":2,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Apr 15 22:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27304,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717848,"statfs":{"total":95995035648,"available":95967076352,"internally_reserved":0,"allocated":360448,"data_stored":139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Apr 15 22:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Apr 15 22:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61199999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.622},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 22:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.309,"15min":0.29799999999999999},"min":{"1min":0.247,"5min":0.193,"15min":0.193},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.25800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.315}]},{"osd":2,"last update":"Mon Apr 15 22:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.247},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27000000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:56:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32200000000000001},"min":{"1min":0.23499999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.316},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.442,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Apr 15 22:56:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Apr 15 22:56:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.35299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T22:56:44.488 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T22:56:44.488 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T22:56:44.488 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-15T22:56:44.489 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-15T22:56:44.706 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:45.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:45 smithi037 ceph-mon[29191]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:56:45.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:45 smithi037 ceph-mon[29191]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:45.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:45 smithi189 ceph-mon[36179]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:56:45.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:45 smithi189 ceph-mon[36179]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:46.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1398768734 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca4100e60 msgr2=0x7f6ca4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1398768734 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca4100e60 0x7f6ca4104e30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009910 tx=0x7f6c94009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1398768734 shutdown_connections 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1398768734 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca4100e60 0x7f6ca4104e30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1398768734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca41004c0 0x7f6ca4100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1398768734 >> 172.21.15.37:0/1398768734 conn(0x7f6ca40fbd80 msgr2=0x7f6ca40fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.383+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1398768734 shutdown_connections 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.384+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1398768734 wait complete. 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.384+0000 7f6ca8f23700 1 Processor -- start 2024-04-15T22:56:46.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.384+0000 7f6ca8f23700 1 -- start start 2024-04-15T22:56:46.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.384+0000 7f6ca8f23700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 0x7f6ca4195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.384+0000 7f6ca8f23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca4100e60 0x7f6ca4196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.384+0000 7f6ca8f23700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca4196b70 con 0x7f6ca41004c0 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca8f23700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca419a900 con 0x7f6ca4100e60 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 0x7f6ca4195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 0x7f6ca4195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32924/0 (socket says 172.21.15.37:32924) 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 -- 172.21.15.37:0/1574011140 learned_addr learned my addr 172.21.15.37:0/1574011140 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca1d9b700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca4100e60 0x7f6ca4196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 -- 172.21.15.37:0/1574011140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca4100e60 msgr2=0x7f6ca4196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca4100e60 0x7f6ca4196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 -- 172.21.15.37:0/1574011140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c940095f0 con 0x7f6ca41004c0 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca259c700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 0x7f6ca4195f50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c8a0 tx=0x7f6c8c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8c00cea0 con 0x7f6ca41004c0 2024-04-15T22:56:46.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca419abe0 con 0x7f6ca41004c0 2024-04-15T22:56:46.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c004830 con 0x7f6ca41004c0 2024-04-15T22:56:46.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8c0055b0 con 0x7f6ca41004c0 2024-04-15T22:56:46.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.385+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca419b130 con 0x7f6ca41004c0 2024-04-15T22:56:46.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.386+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca4108830 con 0x7f6ca41004c0 2024-04-15T22:56:46.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.387+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f6c8c007ad0 con 0x7f6ca41004c0 2024-04-15T22:56:46.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.387+0000 7f6c9b7fe700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6c9006c680 0x7f6c9006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:46.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.387+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6c8c08b340 con 0x7f6ca41004c0 2024-04-15T22:56:46.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.387+0000 7f6ca1d9b700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6c9006c680 0x7f6c9006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:46.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.388+0000 7f6ca1d9b700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6c9006c680 0x7f6c9006eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4197570 tx=0x7f6c9401a040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:46.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.392+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6c8c055cc0 con 0x7f6ca41004c0 2024-04-15T22:56:46.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.635+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6ca404f2a0 con 0x7f6ca41004c0 2024-04-15T22:56:46.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.636+0000 7f6c9b7fe700 1 -- 172.21.15.37:0/1574011140 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6c8c0592e0 con 0x7f6ca41004c0 2024-04-15T22:56:46.628 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:56:46.628 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T22:56:46.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6c9006c680 msgr2=0x7f6c9006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:46.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6c9006c680 0x7f6c9006eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4197570 tx=0x7f6c9401a040 comp rx=0 tx=0).stop 2024-04-15T22:56:46.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 msgr2=0x7f6ca4195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:46.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 0x7f6ca4195f50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c8a0 tx=0x7f6c8c00cc60 comp rx=0 tx=0).stop 2024-04-15T22:56:46.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 shutdown_connections 2024-04-15T22:56:46.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6c9006c680 0x7f6c9006eb30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:46.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ca41004c0 0x7f6ca4195f50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:46.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 --2- 172.21.15.37:0/1574011140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca4100e60 0x7f6ca4196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:46.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 >> 172.21.15.37:0/1574011140 conn(0x7f6ca40fbd80 msgr2=0x7f6ca40fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:46.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 shutdown_connections 2024-04-15T22:56:46.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:46.638+0000 7f6ca8f23700 1 -- 172.21.15.37:0/1574011140 wait complete. 2024-04-15T22:56:47.320 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-15T22:56:47.320 INFO:tasks.cephadm:Setup complete, yielding 2024-04-15T22:56:47.321 INFO:teuthology.run_tasks:Running task print... 2024-04-15T22:56:47.330 INFO:teuthology.task.print:**** done end installing v18.2.0 cephadm ... 2024-04-15T22:56:47.330 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:56:47.393 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:56:47.394 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-15T22:56:47.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:47 smithi037 ceph-mon[29191]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:47.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:47 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1574011140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:56:47.608 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:47.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:47 smithi189 ceph-mon[36179]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:47.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:47 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1574011140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:56:48.752 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:48 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:48.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:48 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 -- 172.21.15.37:0/1022138946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 msgr2=0x7f00840ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 --2- 172.21.15.37:0/1022138946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f00840ff560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0074009910 tx=0x7f0074009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 -- 172.21.15.37:0/1022138946 shutdown_connections 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 --2- 172.21.15.37:0/1022138946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00841003f0 0x7f0084100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 --2- 172.21.15.37:0/1022138946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f00840ff560 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 -- 172.21.15.37:0/1022138946 >> 172.21.15.37:0/1022138946 conn(0x7f00840fa7a0 msgr2=0x7f00840fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 -- 172.21.15.37:0/1022138946 shutdown_connections 2024-04-15T22:56:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 -- 172.21.15.37:0/1022138946 wait complete. 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 Processor -- start 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.221+0000 7f008a283700 1 -- start start 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f008a283700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f0084195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f008a283700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00841003f0 0x7f0084196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f008a283700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084196970 con 0x7f00840ff150 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f008a283700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084196ab0 con 0x7f00841003f0 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f0084195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f0084195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32954/0 (socket says 172.21.15.37:32954) 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 -- 172.21.15.37:0/90080839 learned_addr learned my addr 172.21.15.37:0/90080839 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f00837fe700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00841003f0 0x7f0084196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 -- 172.21.15.37:0/90080839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00841003f0 msgr2=0x7f0084196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00841003f0 0x7f0084196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 -- 172.21.15.37:0/90080839 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740095f0 con 0x7f00840ff150 2024-04-15T22:56:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.222+0000 7f0083fff700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f0084195e10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0074006010 tx=0x7f007400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.223+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f007401d070 con 0x7f00840ff150 2024-04-15T22:56:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.223+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008419b500 con 0x7f00840ff150 2024-04-15T22:56:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.223+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f007400f460 con 0x7f00840ff150 2024-04-15T22:56:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.223+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0074021640 con 0x7f00840ff150 2024-04-15T22:56:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.223+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008419b9f0 con 0x7f00840ff150 2024-04-15T22:56:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.224+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0084068ab0 con 0x7f00840ff150 2024-04-15T22:56:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.224+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f007400fab0 con 0x7f00840ff150 2024-04-15T22:56:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.225+0000 7f00817fa700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f006c06c680 0x7f006c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.225+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f007408c980 con 0x7f00840ff150 2024-04-15T22:56:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.225+0000 7f00837fe700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f006c06c680 0x7f006c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.225+0000 7f00837fe700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f006c06c680 0x7f006c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0078005fd0 tx=0x7f007800a560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:49.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.229+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f007405ae30 con 0x7f00840ff150 2024-04-15T22:56:49.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.427+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f008419bfe0 con 0x7f00840ff150 2024-04-15T22:56:49.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.432+0000 7f00817fa700 1 -- 172.21.15.37:0/90080839 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f00740260d0 con 0x7f00840ff150 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.434+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f006c06c680 msgr2=0x7f006c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.434+0000 7f008a283700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f006c06c680 0x7f006c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0078005fd0 tx=0x7f007800a560 comp rx=0 tx=0).stop 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.434+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 msgr2=0x7f0084195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.434+0000 7f008a283700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f0084195e10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0074006010 tx=0x7f007400be20 comp rx=0 tx=0).stop 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 shutdown_connections 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f006c06c680 0x7f006c06eb30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00840ff150 0x7f0084195e10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 --2- 172.21.15.37:0/90080839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00841003f0 0x7f0084196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:49.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 >> 172.21.15.37:0/90080839 conn(0x7f00840fa7a0 msgr2=0x7f0084068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:49.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 shutdown_connections 2024-04-15T22:56:49.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:49.435+0000 7f008a283700 1 -- 172.21.15.37:0/90080839 wait complete. 2024-04-15T22:56:49.752 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:49 smithi037 ceph-mon[29191]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:49.752 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:49 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/90080839' entity='client.admin' 2024-04-15T22:56:49.752 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:56:49.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:49 smithi189 ceph-mon[36179]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:49.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:49 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/90080839' entity='client.admin' 2024-04-15T22:56:49.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:56:50.109 INFO:teuthology.run_tasks:Running task print... 2024-04-15T22:56:50.118 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-15T22:56:50.118 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:56:50.171 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:56:50.171 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch status' 2024-04-15T22:56:50.390 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:50.766 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:50 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:56:50.766 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:50 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:56:50.766 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:50 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:56:50.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:50 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:56:50.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:50 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:56:50.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:50 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:56:51.805 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:51 smithi037 ceph-mon[29191]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:51.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:51 smithi189 ceph-mon[36179]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:52.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 -- 172.21.15.37:0/2864596393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa4100350 msgr2=0x7f0aa41007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:52.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/2864596393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa4100350 0x7f0aa41007a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009910 tx=0x7f0a94009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:52.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 -- 172.21.15.37:0/2864596393 shutdown_connections 2024-04-15T22:56:52.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/2864596393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa4100350 0x7f0aa41007a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:52.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/2864596393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aa40ff150 0x7f0aa40ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:52.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 -- 172.21.15.37:0/2864596393 >> 172.21.15.37:0/2864596393 conn(0x7f0aa40fa700 msgr2=0x7f0aa40fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:52.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 -- 172.21.15.37:0/2864596393 shutdown_connections 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.054+0000 7f0aabf0b700 1 -- 172.21.15.37:0/2864596393 wait complete. 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aabf0b700 1 Processor -- start 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aabf0b700 1 -- start start 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aabf0b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 0x7f0aa4195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aabf0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aa4100350 0x7f0aa4196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aabf0b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41966a0 con 0x7f0aa40ff150 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aabf0b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41967e0 con 0x7f0aa4100350 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aa9ca7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 0x7f0aa4195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aa9ca7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 0x7f0aa4195b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33480/0 (socket says 172.21.15.37:33480) 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.055+0000 7f0aa9ca7700 1 -- 172.21.15.37:0/911029487 learned_addr learned my addr 172.21.15.37:0/911029487 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aa94a6700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aa4100350 0x7f0aa4196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aa9ca7700 1 -- 172.21.15.37:0/911029487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aa4100350 msgr2=0x7f0aa4196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:52.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aa9ca7700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aa4100350 0x7f0aa4196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aa9ca7700 1 -- 172.21.15.37:0/911029487 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a940095f0 con 0x7f0aa40ff150 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aa9ca7700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 0x7f0aa4195b40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000b9c0 tx=0x7f0aa000bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa000c980 con 0x7f0aa40ff150 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0aa0014440 con 0x7f0aa40ff150 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa419b290 con 0x7f0aa40ff150 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa0012550 con 0x7f0aa40ff150 2024-04-15T22:56:52.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.056+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa419b7e0 con 0x7f0aa40ff150 2024-04-15T22:56:52.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.057+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa4107d40 con 0x7f0aa40ff150 2024-04-15T22:56:52.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.057+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0aa00145b0 con 0x7f0aa40ff150 2024-04-15T22:56:52.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.058+0000 7f0a9affd700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0a9006c680 0x7f0a9006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:52.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.058+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0aa0006e40 con 0x7f0aa40ff150 2024-04-15T22:56:52.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.058+0000 7f0aa94a6700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0a9006c680 0x7f0a9006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:52.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.059+0000 7f0aa94a6700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0a9006c680 0x7f0a9006eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a940059d0 tx=0x7f0a94005920 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:52.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.063+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0aa004e720 con 0x7f0aa40ff150 2024-04-15T22:56:52.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.282+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0aa419bed0 con 0x7f0a9006c680 2024-04-15T22:56:52.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.283+0000 7f0a9affd700 1 -- 172.21.15.37:0/911029487 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0aa419bed0 con 0x7f0a9006c680 2024-04-15T22:56:52.276 INFO:teuthology.orchestra.run.smithi037.stdout:Backend: cephadm 2024-04-15T22:56:52.276 INFO:teuthology.orchestra.run.smithi037.stdout:Available: Yes 2024-04-15T22:56:52.276 INFO:teuthology.orchestra.run.smithi037.stdout:Paused: No 2024-04-15T22:56:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0a9006c680 msgr2=0x7f0a9006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0a9006c680 0x7f0a9006eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a940059d0 tx=0x7f0a94005920 comp rx=0 tx=0).stop 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 msgr2=0x7f0aa4195b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 0x7f0aa4195b40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000b9c0 tx=0x7f0aa000bd80 comp rx=0 tx=0).stop 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 shutdown_connections 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0a9006c680 0x7f0a9006eb30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0aa40ff150 0x7f0aa4195b40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 --2- 172.21.15.37:0/911029487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aa4100350 0x7f0aa4196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 >> 172.21.15.37:0/911029487 conn(0x7f0aa40fa700 msgr2=0x7f0aa4068df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 shutdown_connections 2024-04-15T22:56:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:52.286+0000 7f0aabf0b700 1 -- 172.21.15.37:0/911029487 wait complete. 2024-04-15T22:56:52.881 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-15T22:56:53.097 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:53.454 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:53 smithi037 ceph-mon[29191]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:56:53.454 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:53 smithi037 ceph-mon[29191]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:53.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:53 smithi189 ceph-mon[36179]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:56:53.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:53 smithi189 ceph-mon[36179]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:54.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 -- 172.21.15.37:0/3345118637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f01003d0 msgr2=0x7f39f0100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 --2- 172.21.15.37:0/3345118637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f01003d0 0x7f39f0100840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009910 tx=0x7f39e4009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 -- 172.21.15.37:0/3345118637 shutdown_connections 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 --2- 172.21.15.37:0/3345118637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f01003d0 0x7f39f0100840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 --2- 172.21.15.37:0/3345118637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f00ff130 0x7f39f00ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 -- 172.21.15.37:0/3345118637 >> 172.21.15.37:0/3345118637 conn(0x7f39f00fa780 msgr2=0x7f39f00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 -- 172.21.15.37:0/3345118637 shutdown_connections 2024-04-15T22:56:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.700+0000 7f39f4996700 1 -- 172.21.15.37:0/3345118637 wait complete. 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39f4996700 1 Processor -- start 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39f4996700 1 -- start start 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39f4996700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 0x7f39f0103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39f4996700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f01003d0 0x7f39f01017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39f4996700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f0101cf0 con 0x7f39f00ff130 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39f4996700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f0101e30 con 0x7f39f01003d0 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39ee59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 0x7f39f0103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39ee59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 0x7f39f0103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33504/0 (socket says 172.21.15.37:33504) 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.701+0000 7f39ee59c700 1 -- 172.21.15.37:0/1006255249 learned_addr learned my addr 172.21.15.37:0/1006255249 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39edd9b700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f01003d0 0x7f39f01017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39ee59c700 1 -- 172.21.15.37:0/1006255249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f01003d0 msgr2=0x7f39f01017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39ee59c700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f01003d0 0x7f39f01017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39ee59c700 1 -- 172.21.15.37:0/1006255249 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e40095f0 con 0x7f39f00ff130 2024-04-15T22:56:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39edd9b700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f01003d0 0x7f39f01017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T22:56:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39ee59c700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 0x7f39f0103160 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f39e000b4b0 tx=0x7f39e000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e0011800 con 0x7f39f00ff130 2024-04-15T22:56:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f0102110 con 0x7f39f00ff130 2024-04-15T22:56:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f39e0011e10 con 0x7f39f00ff130 2024-04-15T22:56:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e000f2e0 con 0x7f39f00ff130 2024-04-15T22:56:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.702+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f019db40 con 0x7f39f00ff130 2024-04-15T22:56:54.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.703+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f0107c90 con 0x7f39f00ff130 2024-04-15T22:56:54.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.703+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f39e0010380 con 0x7f39f00ff130 2024-04-15T22:56:54.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.704+0000 7f39df7fe700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f39d806c680 0x7f39d806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:54.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.704+0000 7f39edd9b700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f39d806c680 0x7f39d806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:54.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.704+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f39e008b1d0 con 0x7f39f00ff130 2024-04-15T22:56:54.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.704+0000 7f39edd9b700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f39d806c680 0x7f39d806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39e400b5c0 tx=0x7f39e4005e20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.709+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f39e0059680 con 0x7f39f00ff130 2024-04-15T22:56:54.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.916+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f39f0068c50 con 0x7f39d806c680 2024-04-15T22:56:54.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.926+0000 7f39df7fe700 1 -- 172.21.15.37:0/1006255249 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f39f0068c50 con 0x7f39d806c680 2024-04-15T22:56:54.918 INFO:teuthology.orchestra.run.smithi037.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:56:54.918 INFO:teuthology.orchestra.run.smithi037.stdout:alertmanager.smithi037 smithi037 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 e049eff81a04 2024-04-15T22:56:54.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter.smithi037 smithi037 running (5m) 2m ago 5m 7063k - 18.2.0 dc2bc1663786 3616d122dfb5 2024-04-15T22:56:54.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter.smithi189 smithi189 running (4m) 64s ago 4m 6966k - 18.2.0 dc2bc1663786 4b1aba4b2eba 2024-04-15T22:56:54.918 INFO:teuthology.orchestra.run.smithi037.stdout:crash.smithi037 smithi037 running (5m) 2m ago 5m 6665k - 18.2.0 dc2bc1663786 944063fe7d85 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:crash.smithi189 smithi189 running (4m) 64s ago 4m 6660k - 18.2.0 dc2bc1663786 f483fc8d01e0 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:grafana.smithi037 smithi037 *:3000 running (3m) 2m ago 5m 78.2M - 9.4.7 954c08fa6188 a7a1c1b0e042 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.smithi037.gtwkck smithi037 *:9283,8765,8443 running (7m) 2m ago 7m 478M - 18.2.0 dc2bc1663786 9fa350a65979 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.smithi189.lanqnp smithi189 *:8443,9283,8765 running (4m) 64s ago 4m 437M - 18.2.0 dc2bc1663786 a9d94b5f083f 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:mon.smithi037 smithi037 running (7m) 2m ago 7m 46.6M 2048M 18.2.0 dc2bc1663786 f39d1d96ec93 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:mon.smithi189 smithi189 running (4m) 64s ago 4m 40.6M 2048M 18.2.0 dc2bc1663786 e2331fdccbaa 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter.smithi037 smithi037 *:9100 running (5m) 2m ago 5m 16.1M - 1.5.0 0da6a335fe13 a4d1b4d06e69 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter.smithi189 smithi189 *:9100 running (4m) 64s ago 4m 15.1M - 1.5.0 0da6a335fe13 f9d0ca18f1f6 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 smithi037 running (3m) 2m ago 3m 40.9M 3948M 18.2.0 dc2bc1663786 45282f17f939 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 smithi037 running (2m) 2m ago 2m 44.8M 3948M 18.2.0 dc2bc1663786 7c4e631bf214 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 smithi037 running (2m) 2m ago 2m 42.4M 3948M 18.2.0 dc2bc1663786 2c1c43054c1a 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 smithi189 running (2m) 64s ago 2m 41.9M 4970M 18.2.0 dc2bc1663786 5e54e0afe09f 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 smithi189 running (96s) 64s ago 96s 40.0M 4970M 18.2.0 dc2bc1663786 c6102fb44c70 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 smithi189 running (70s) 64s ago 69s 14.3M 4970M 18.2.0 dc2bc1663786 e4f59c36ac6f 2024-04-15T22:56:54.919 INFO:teuthology.orchestra.run.smithi037.stdout:prometheus.smithi037 smithi037 *:9095 running (3m) 2m ago 4m 40.3M - 2.43.0 a07b618ecd1d a2075b4529f9 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f39d806c680 msgr2=0x7f39d806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f39d806c680 0x7f39d806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39e400b5c0 tx=0x7f39e4005e20 comp rx=0 tx=0).stop 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 msgr2=0x7f39f0103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 0x7f39f0103160 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f39e000b4b0 tx=0x7f39e000b870 comp rx=0 tx=0).stop 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 shutdown_connections 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f39d806c680 0x7f39d806eb30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39f00ff130 0x7f39f0103160 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 --2- 172.21.15.37:0/1006255249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f01003d0 0x7f39f01017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.928+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 >> 172.21.15.37:0/1006255249 conn(0x7f39f00fa780 msgr2=0x7f39f0068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.929+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 shutdown_connections 2024-04-15T22:56:54.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:54.929+0000 7f39f4996700 1 -- 172.21.15.37:0/1006255249 wait complete. 2024-04-15T22:56:55.531 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:55 smithi037 ceph-mon[29191]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:55.552 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-15T22:56:55.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:55 smithi189 ceph-mon[36179]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:55.764 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:56.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:56 smithi189 ceph-mon[36179]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:56:56.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:56 smithi037 ceph-mon[29191]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 -- 172.21.15.37:0/911824924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 msgr2=0x7ff32c0ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 --2- 172.21.15.37:0/911824924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c0ff930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff314009910 tx=0x7ff314009c20 comp rx=0 tx=0).stop 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 -- 172.21.15.37:0/911824924 shutdown_connections 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 --2- 172.21.15.37:0/911824924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c0ff930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 --2- 172.21.15.37:0/911824924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0feb70 0x7ff32c0fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 -- 172.21.15.37:0/911824924 >> 172.21.15.37:0/911824924 conn(0x7ff32c0fa7a0 msgr2=0x7ff32c0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 -- 172.21.15.37:0/911824924 shutdown_connections 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 -- 172.21.15.37:0/911824924 wait complete. 2024-04-15T22:56:57.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.412+0000 7ff3322ac700 1 Processor -- start 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff3322ac700 1 -- start start 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff3322ac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0feb70 0x7ff32c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff3322ac700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff3322ac700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c198b70 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff3322ac700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c198cb0 con 0x7ff32c0feb70 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33520/0 (socket says 172.21.15.37:33520) 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 -- 172.21.15.37:0/3645367853 learned_addr learned my addr 172.21.15.37:0/3645367853 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff32bfff700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0feb70 0x7ff32c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 -- 172.21.15.37:0/3645367853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0feb70 msgr2=0x7ff32c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0feb70 0x7ff32c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 -- 172.21.15.37:0/3645367853 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140095f0 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.413+0000 7ff323fff700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c198550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff31400b5c0 tx=0x7ff31400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:57.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.414+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31401d070 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.414+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff314022470 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.414+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c19d700 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.414+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31400f690 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.414+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c19dbf0 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.415+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c068ab0 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.415+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff3140225e0 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.415+0000 7ff329ffb700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff31806c680 0x7ff31806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:56:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.416+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff31408cd10 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.416+0000 7ff32bfff700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff31806c680 0x7ff31806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:56:57.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.416+0000 7ff32bfff700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff31806c680 0x7ff31806eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff32c069890 tx=0x7ff31c009190 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:56:57.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.420+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff31405b1c0 con 0x7ff32c0ff4c0 2024-04-15T22:56:57.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:57 smithi037 ceph-mon[29191]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.629+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff32c1089c0 con 0x7ff31806c680 2024-04-15T22:56:57.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.633+0000 7ff329ffb700 1 -- 172.21.15.37:0/3645367853 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7ff32c1089c0 con 0x7ff31806c680 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:crash 2/2 2m ago 6m * 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:mgr 2/2 2m ago 6m count:2 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:mon 2/2 2m ago 5m smithi037:172.21.15.37=smithi037;smithi189:172.21.15.189=smithi189;count:2 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:osd 6 2m ago - 2024-04-15T22:56:57.626 INFO:teuthology.orchestra.run.smithi037.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff31806c680 msgr2=0x7ff31806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff31806c680 0x7ff31806eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff32c069890 tx=0x7ff31c009190 comp rx=0 tx=0).stop 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 msgr2=0x7ff32c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c198550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff31400b5c0 tx=0x7ff31400bd70 comp rx=0 tx=0).stop 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 shutdown_connections 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7ff31806c680 0x7ff31806eb30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:57.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0feb70 0x7ff32c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:57.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 --2- 172.21.15.37:0/3645367853 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff32c0ff4c0 0x7ff32c198550 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:56:57.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.635+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 >> 172.21.15.37:0/3645367853 conn(0x7ff32c0fa7a0 msgr2=0x7ff32c107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:56:57.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.636+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 shutdown_connections 2024-04-15T22:56:57.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:56:57.636+0000 7ff3322ac700 1 -- 172.21.15.37:0/3645367853 wait complete. 2024-04-15T22:56:57.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:57 smithi189 ceph-mon[36179]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:58.294 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-15T22:56:58.510 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:56:59.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:59 smithi189 ceph-mon[36179]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:56:59.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:56:59 smithi189 ceph-mon[36179]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:56:59.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:59 smithi037 ceph-mon[29191]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:56:59.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:56:59 smithi037 ceph-mon[29191]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:00.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 -- 172.21.15.37:0/1266605121 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b80ff130 msgr2=0x7fe4b80ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:00.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 --2- 172.21.15.37:0/1266605121 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b80ff130 0x7fe4b80ff540 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009910 tx=0x7fe4ac009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 -- 172.21.15.37:0/1266605121 shutdown_connections 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 --2- 172.21.15.37:0/1266605121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4b81003d0 0x7fe4b8100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 --2- 172.21.15.37:0/1266605121 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b80ff130 0x7fe4b80ff540 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 -- 172.21.15.37:0/1266605121 >> 172.21.15.37:0/1266605121 conn(0x7fe4b80fa780 msgr2=0x7fe4b80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 -- 172.21.15.37:0/1266605121 shutdown_connections 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.195+0000 7fe4bee83700 1 -- 172.21.15.37:0/1266605121 wait complete. 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bee83700 1 Processor -- start 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bee83700 1 -- start start 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bee83700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4b80ff130 0x7fe4b80690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bee83700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 0x7fe4b819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bee83700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b819ade0 con 0x7fe4b81003d0 2024-04-15T22:57:00.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bee83700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b819af20 con 0x7fe4b80ff130 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4b7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 0x7fe4b819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4b7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 0x7fe4b819a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57104/0 (socket says 172.21.15.37:57104) 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4b7fff700 1 -- 172.21.15.37:0/601871131 learned_addr learned my addr 172.21.15.37:0/601871131 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4bcc1f700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4b80ff130 0x7fe4b80690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4b7fff700 1 -- 172.21.15.37:0/601871131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4b80ff130 msgr2=0x7fe4b80690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4b7fff700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4b80ff130 0x7fe4b80690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.196+0000 7fe4b7fff700 1 -- 172.21.15.37:0/601871131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0095f0 con 0x7fe4b81003d0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.197+0000 7fe4b7fff700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 0x7fe4b819a810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400d8d0 tx=0x7fe4a400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.197+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4a40098a0 con 0x7fe4b81003d0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.197+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe4a4010460 con 0x7fe4b81003d0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.197+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b819b200 con 0x7fe4b81003d0 2024-04-15T22:57:00.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.197+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4a400f5f0 con 0x7fe4b81003d0 2024-04-15T22:57:00.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.197+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b819b750 con 0x7fe4b81003d0 2024-04-15T22:57:00.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.198+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b8107c10 con 0x7fe4b81003d0 2024-04-15T22:57:00.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.198+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe4a4009a00 con 0x7fe4b81003d0 2024-04-15T22:57:00.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.199+0000 7fe4b5ffb700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4a806c680 0x7fe4a806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:00.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.199+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe4a408b320 con 0x7fe4b81003d0 2024-04-15T22:57:00.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.199+0000 7fe4bcc1f700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4a806c680 0x7fe4a806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:00.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.199+0000 7fe4bcc1f700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4a806c680 0x7fe4a806eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac006010 tx=0x7fe4ac00b540 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:00.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.203+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe4a4059850 con 0x7fe4b81003d0 2024-04-15T22:57:00.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.409+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4b8104d20 con 0x7fe4a806c680 2024-04-15T22:57:00.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.410+0000 7fe4b5ffb700 1 -- 172.21.15.37:0/601871131 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fe4b8104d20 con 0x7fe4a806c680 2024-04-15T22:57:00.402 INFO:teuthology.orchestra.run.smithi037.stdout:HOST ADDR LABELS STATUS 2024-04-15T22:57:00.402 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 172.21.15.37 2024-04-15T22:57:00.403 INFO:teuthology.orchestra.run.smithi037.stdout:smithi189 172.21.15.189 2024-04-15T22:57:00.403 INFO:teuthology.orchestra.run.smithi037.stdout:2 hosts in cluster 2024-04-15T22:57:00.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4a806c680 msgr2=0x7fe4a806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:00.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4a806c680 0x7fe4a806eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac006010 tx=0x7fe4ac00b540 comp rx=0 tx=0).stop 2024-04-15T22:57:00.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 msgr2=0x7fe4b819a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 0x7fe4b819a810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400d8d0 tx=0x7fe4a400dbe0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 shutdown_connections 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe4a806c680 0x7fe4a806eb30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4b80ff130 0x7fe4b80690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 --2- 172.21.15.37:0/601871131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4b81003d0 0x7fe4b819a810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 >> 172.21.15.37:0/601871131 conn(0x7fe4b80fa780 msgr2=0x7fe4b8103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 shutdown_connections 2024-04-15T22:57:00.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:00.413+0000 7fe4bee83700 1 -- 172.21.15.37:0/601871131 wait complete. 2024-04-15T22:57:01.022 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-15T22:57:01.233 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:01.661 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:01 smithi037 ceph-mon[29191]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:01.661 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:01 smithi037 ceph-mon[29191]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:57:01.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:01 smithi189 ceph-mon[36179]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:01.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:01 smithi189 ceph-mon[36179]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:57:02.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 -- 172.21.15.37:0/1209933071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a4100d80 msgr2=0x7f41a4103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 --2- 172.21.15.37:0/1209933071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a4100d80 0x7f41a4103160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4194009910 tx=0x7f4194009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 -- 172.21.15.37:0/1209933071 shutdown_connections 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 --2- 172.21.15.37:0/1209933071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a41036a0 0x7f41a4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 --2- 172.21.15.37:0/1209933071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a4100d80 0x7f41a4103160 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 -- 172.21.15.37:0/1209933071 >> 172.21.15.37:0/1209933071 conn(0x7f41a40fa7a0 msgr2=0x7f41a40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 -- 172.21.15.37:0/1209933071 shutdown_connections 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 -- 172.21.15.37:0/1209933071 wait complete. 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 Processor -- start 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.783+0000 7f41aa085700 1 -- start start 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41aa085700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a4100d80 0x7f41a41006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41aa085700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 0x7f41a40fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41aa085700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a40ff270 con 0x7f41a41036a0 2024-04-15T22:57:02.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41aa085700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a40ff3b0 con 0x7f41a4100d80 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 0x7f41a40fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 0x7f41a40fed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57128/0 (socket says 172.21.15.37:57128) 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 -- 172.21.15.37:0/936668740 learned_addr learned my addr 172.21.15.37:0/936668740 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a37fe700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a4100d80 0x7f41a41006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 -- 172.21.15.37:0/936668740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a4100d80 msgr2=0x7f41a41006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a4100d80 0x7f41a41006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 -- 172.21.15.37:0/936668740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940095f0 con 0x7f41a41036a0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a2ffd700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 0x7f41a40fed30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f419800d8d0 tx=0x7f419800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41980098a0 con 0x7f41a41036a0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4198010460 con 0x7f41a41036a0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a40ff690 con 0x7f41a41036a0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f419800f5f0 con 0x7f41a41036a0 2024-04-15T22:57:02.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.784+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a419de50 con 0x7f41a41036a0 2024-04-15T22:57:02.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.785+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a418fbe0 con 0x7f41a41036a0 2024-04-15T22:57:02.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.786+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f4198009a00 con 0x7f41a41036a0 2024-04-15T22:57:02.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.786+0000 7f41a0ff9700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f418c06c680 0x7f418c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:02.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.786+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f419808b420 con 0x7f41a41036a0 2024-04-15T22:57:02.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.786+0000 7f41a37fe700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f418c06c680 0x7f418c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.787+0000 7f41a37fe700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f418c06c680 0x7f418c06eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4194005090 tx=0x7f4194005a90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.791+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f41980598d0 con 0x7f41a41036a0 2024-04-15T22:57:02.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.998+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f41a402cf10 con 0x7f418c06c680 2024-04-15T22:57:02.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:02.999+0000 7f41a0ff9700 1 -- 172.21.15.37:0/936668740 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f41a402cf10 con 0x7f418c06c680 2024-04-15T22:57:02.991 INFO:teuthology.orchestra.run.smithi037.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-15T22:57:02.992 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005T400BGN 372G No 2m ago LVM detected, locked 2024-04-15T22:57:02.992 INFO:teuthology.orchestra.run.smithi037.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 69s ago LVM detected, locked 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f418c06c680 msgr2=0x7f418c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f418c06c680 0x7f418c06eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4194005090 tx=0x7f4194005a90 comp rx=0 tx=0).stop 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 msgr2=0x7f41a40fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 0x7f41a40fed30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f419800d8d0 tx=0x7f419800dbe0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 shutdown_connections 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f418c06c680 0x7f418c06eb30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a4100d80 0x7f41a41006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 --2- 172.21.15.37:0/936668740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41a41036a0 0x7f41a40fed30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:02.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.001+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 >> 172.21.15.37:0/936668740 conn(0x7f41a40fa7a0 msgr2=0x7f41a40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:02.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.002+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 shutdown_connections 2024-04-15T22:57:02.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:03.002+0000 7f41aa085700 1 -- 172.21.15.37:0/936668740 wait complete. 2024-04-15T22:57:03.629 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:03 smithi037 ceph-mon[29191]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:03.650 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:57:03.659 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:03.660 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-15T22:57:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:03 smithi189 ceph-mon[36179]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:04.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:04 smithi037 ceph-mon[29191]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:57:04.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:04.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:04 smithi189 ceph-mon[36179]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:57:04.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:05.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 -- 172.21.15.37:0/3521885016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 msgr2=0x7f28f01018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3521885016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01018a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009910 tx=0x7f28e4009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 -- 172.21.15.37:0/3521885016 shutdown_connections 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3521885016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01018a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3521885016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f0100250 0x7f28f0100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 -- 172.21.15.37:0/3521885016 >> 172.21.15.37:0/3521885016 conn(0x7f28f00fb7e0 msgr2=0x7f28f00fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 -- 172.21.15.37:0/3521885016 shutdown_connections 2024-04-15T22:57:05.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.563+0000 7f28f84fb700 1 -- 172.21.15.37:0/3521885016 wait complete. 2024-04-15T22:57:05.556 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:05 smithi037 ceph-mon[29191]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f84fb700 1 Processor -- start 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f84fb700 1 -- start start 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f84fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f0100250 0x7f28f0197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f84fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f84fb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f01987c0 con 0x7f28f0101450 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f84fb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0198930 con 0x7f28f0100250 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f5a96700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f5a96700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57146/0 (socket says 172.21.15.37:57146) 2024-04-15T22:57:05.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f5a96700 1 -- 172.21.15.37:0/3586437799 learned_addr learned my addr 172.21.15.37:0/3586437799 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f6297700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f0100250 0x7f28f0197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f5a96700 1 -- 172.21.15.37:0/3586437799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f0100250 msgr2=0x7f28f0197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f5a96700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f0100250 0x7f28f0197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.564+0000 7f28f5a96700 1 -- 172.21.15.37:0/3586437799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e40095f0 con 0x7f28f0101450 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.565+0000 7f28f5a96700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01981c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f28e4006010 tx=0x7f28e400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.565+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28e401d070 con 0x7f28f0101450 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.565+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28e400f460 con 0x7f28f0101450 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.565+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f019d360 con 0x7f28f0101450 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.565+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28e4021640 con 0x7f28f0101450 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.565+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f019d850 con 0x7f28f0101450 2024-04-15T22:57:05.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.566+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f0105530 con 0x7f28f0101450 2024-04-15T22:57:05.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.566+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f28e400fab0 con 0x7f28f0101450 2024-04-15T22:57:05.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.567+0000 7f28ef7fe700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f28d806c680 0x7f28d806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:05.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.567+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f28e408c6f0 con 0x7f28f0101450 2024-04-15T22:57:05.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.567+0000 7f28f6297700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f28d806c680 0x7f28d806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:05.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.567+0000 7f28f6297700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f28d806c680 0x7f28d806eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28f0198b60 tx=0x7f28e000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:05.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.571+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f28e4057190 con 0x7f28f0101450 2024-04-15T22:57:05.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:05 smithi189 ceph-mon[36179]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:05.797+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f28f0061190 con 0x7f28d806c680 2024-04-15T22:57:06.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:06 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-15T22:57:06.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:06 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-15T22:57:07.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.423+0000 7f28ef7fe700 1 -- 172.21.15.37:0/3586437799 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f28f0061190 con 0x7f28d806c680 2024-04-15T22:57:07.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.428+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f28d806c680 msgr2=0x7f28d806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:07.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.428+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f28d806c680 0x7f28d806eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28f0198b60 tx=0x7f28e000b410 comp rx=0 tx=0).stop 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 msgr2=0x7f28f01981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01981c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f28e4006010 tx=0x7f28e400bd70 comp rx=0 tx=0).stop 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 shutdown_connections 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f28d806c680 0x7f28d806eb30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f0100250 0x7f28f0197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 --2- 172.21.15.37:0/3586437799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f0101450 0x7f28f01981c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 >> 172.21.15.37:0/3586437799 conn(0x7f28f00fb7e0 msgr2=0x7f28f0102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 shutdown_connections 2024-04-15T22:57:07.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:07.429+0000 7f28f84fb700 1 -- 172.21.15.37:0/3586437799 wait complete. 2024-04-15T22:57:07.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:07 smithi037 ceph-mon[29191]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:57:07.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:07 smithi037 ceph-mon[29191]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:07.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:07 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-15T22:57:07.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:07 smithi037 ceph-mon[29191]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:57:07.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:07 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-15T22:57:07.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:07 smithi037 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi037[29187]: 2024-04-15T22:57:07.401+0000 7fc1a2457700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T22:57:07.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:07 smithi189 ceph-mon[36179]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:57:07.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:07 smithi189 ceph-mon[36179]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:07.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:07 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-15T22:57:07.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:07 smithi189 ceph-mon[36179]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:57:07.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:07 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-15T22:57:08.149 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-15T22:57:08.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: fsmap cephfs:0 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: Saving service mds.cephfs spec with placement count:4 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.rdehay", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.rdehay", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:08.673 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:08 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:08.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: fsmap cephfs:0 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: Saving service mds.cephfs spec with placement count:4 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:08.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.rdehay", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:08.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.rdehay", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:08.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:08 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:08.970 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:09.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:09 smithi189 ceph-mon[36179]: Deploying daemon mds.cephfs.smithi037.rdehay on smithi037 2024-04-15T22:57:09.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:09 smithi189 ceph-mon[36179]: pgmap v176: 65 pgs: 8 creating+peering, 15 active+clean, 42 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:09.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:09 smithi189 ceph-mon[36179]: osdmap e44: 6 total, 6 up, 6 in 2024-04-15T22:57:09.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:09 smithi037 ceph-mon[29191]: Deploying daemon mds.cephfs.smithi037.rdehay on smithi037 2024-04-15T22:57:09.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:09 smithi037 ceph-mon[29191]: pgmap v176: 65 pgs: 8 creating+peering, 15 active+clean, 42 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:09.816 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:09 smithi037 ceph-mon[29191]: osdmap e44: 6 total, 6 up, 6 in 2024-04-15T22:57:10.528 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:10 smithi037 ceph-mon[29191]: osdmap e45: 6 total, 6 up, 6 in 2024-04-15T22:57:10.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:10 smithi189 ceph-mon[36179]: osdmap e45: 6 total, 6 up, 6 in 2024-04-15T22:57:11.505 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:11 smithi037 ceph-mon[29191]: pgmap v179: 65 pgs: 8 creating+peering, 51 active+clean, 6 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 -- 172.21.15.37:0/317202132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 msgr2=0x7fa17c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 --2- 172.21.15.37:0/317202132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c100650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009910 tx=0x7fa16c009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 -- 172.21.15.37:0/317202132 shutdown_connections 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 --2- 172.21.15.37:0/317202132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa17c101440 0x7fa17c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 --2- 172.21.15.37:0/317202132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c100650 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 -- 172.21.15.37:0/317202132 >> 172.21.15.37:0/317202132 conn(0x7fa17c0fb7f0 msgr2=0x7fa17c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 -- 172.21.15.37:0/317202132 shutdown_connections 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 -- 172.21.15.37:0/317202132 wait complete. 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 Processor -- start 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.535+0000 7fa1818e4700 1 -- start start 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1818e4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1818e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa17c101440 0x7fa17c196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:11.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1818e4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c196630 con 0x7fa17c100240 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1818e4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c196770 con 0x7fa17c101440 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41638/0 (socket says 172.21.15.37:41638) 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 -- 172.21.15.37:0/3901401474 learned_addr learned my addr 172.21.15.37:0/3901401474 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1727fc700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa17c101440 0x7fa17c196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 -- 172.21.15.37:0/3901401474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa17c101440 msgr2=0x7fa17c196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa17c101440 0x7fa17c196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 -- 172.21.15.37:0/3901401474 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c0095f0 con 0x7fa17c100240 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa17affd700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c195ad0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00bda0 tx=0x7fa16c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa16c01d070 con 0x7fa17c100240 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa16c004c50 con 0x7fa17c100240 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c19b1c0 con 0x7fa17c100240 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.536+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa16c00f690 con 0x7fa17c100240 2024-04-15T22:57:11.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.537+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c19b6b0 con 0x7fa17c100240 2024-04-15T22:57:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.537+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c108d90 con 0x7fa17c100240 2024-04-15T22:57:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.538+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa16c0229b0 con 0x7fa17c100240 2024-04-15T22:57:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.538+0000 7fa1808e2700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa16806c680 0x7fa16806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.539+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fa16c08cef0 con 0x7fa17c100240 2024-04-15T22:57:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.539+0000 7fa1727fc700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa16806c680 0x7fa16806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.539+0000 7fa1727fc700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa16806c680 0x7fa16806eb30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa164005fd0 tx=0x7fa16400a560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:11.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.543+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa16c05b110 con 0x7fa17c100240 2024-04-15T22:57:11.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:11 smithi189 ceph-mon[36179]: pgmap v179: 65 pgs: 8 creating+peering, 51 active+clean, 6 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.802+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa17c108f80 con 0x7fa17c100240 2024-04-15T22:57:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.802+0000 7fa1808e2700 1 -- 172.21.15.37:0/3901401474 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fa16c027460 con 0x7fa17c100240 2024-04-15T22:57:11.795 INFO:teuthology.orchestra.run.smithi037.stdout:e2 2024-04-15T22:57:11.795 INFO:teuthology.orchestra.run.smithi037.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:57:11.795 INFO:teuthology.orchestra.run.smithi037.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:57:11.795 INFO:teuthology.orchestra.run.smithi037.stdout:legacy client fscid: 1 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:Filesystem 'cephfs' (1) 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:fs_name cephfs 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 2 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-04-15T22:57:07.400892+0000 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:modified 2024-04-15T22:57:07.402726+0000 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:tableserver 0 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:root 0 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:session_timeout 60 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:session_autoclose 300 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:max_file_size 1099511627776 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:required_client_features {} 2024-04-15T22:57:11.796 INFO:teuthology.orchestra.run.smithi037.stdout:last_failure 0 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:last_failure_osd_epoch 0 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:max_mds 1 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:in 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:up {} 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:failed 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:damaged 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:stopped 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:data_pools [3] 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:metadata_pool 2 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:inline_data disabled 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:balancer 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:bal_rank_mask -1 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout:standby_count_wanted 0 2024-04-15T22:57:11.797 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:11.798 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:11.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa16806c680 msgr2=0x7fa16806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:11.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa16806c680 0x7fa16806eb30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa164005fd0 tx=0x7fa16400a560 comp rx=0 tx=0).stop 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 msgr2=0x7fa17c195ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c195ad0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00bda0 tx=0x7fa16c00be80 comp rx=0 tx=0).stop 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 shutdown_connections 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa16806c680 0x7fa16806eb30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c100240 0x7fa17c195ad0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 --2- 172.21.15.37:0/3901401474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa17c101440 0x7fa17c196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 >> 172.21.15.37:0/3901401474 conn(0x7fa17c0fb7f0 msgr2=0x7fa17c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 shutdown_connections 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:11.804+0000 7fa1818e4700 1 -- 172.21.15.37:0/3901401474 wait complete. 2024-04-15T22:57:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 2 2024-04-15T22:57:12.432 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:57:12.441 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:12.442 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-15T22:57:12.653 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:12.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:12.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:12.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:12.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.pqjbmb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:12.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.pqjbmb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: Deploying daemon mds.cephfs.smithi189.pqjbmb on smithi189 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3901401474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: daemon mds.cephfs.smithi037.rdehay assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: Cluster is now healthy 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6826/479125410,v1:172.21.15.37:6827/479125410] up:boot 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:creating} 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi037.rdehay"}]: dispatch 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: daemon mds.cephfs.smithi037.rdehay is now active in filesystem cephfs as rank 0 2024-04-15T22:57:12.963 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:12 smithi189 ceph-mon[36179]: pgmap v180: 65 pgs: 8 creating+peering, 57 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.pqjbmb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.pqjbmb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: Deploying daemon mds.cephfs.smithi189.pqjbmb on smithi189 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3901401474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: daemon mds.cephfs.smithi037.rdehay assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-15T22:57:12.997 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: Cluster is now healthy 2024-04-15T22:57:12.998 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6826/479125410,v1:172.21.15.37:6827/479125410] up:boot 2024-04-15T22:57:12.998 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:creating} 2024-04-15T22:57:12.998 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi037.rdehay"}]: dispatch 2024-04-15T22:57:12.998 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: daemon mds.cephfs.smithi037.rdehay is now active in filesystem cephfs as rank 0 2024-04-15T22:57:12.998 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:12 smithi037 ceph-mon[29191]: pgmap v180: 65 pgs: 8 creating+peering, 57 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:57:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 -- 172.21.15.37:0/1368415524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948100240 msgr2=0x7f8948100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 --2- 172.21.15.37:0/1368415524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948100240 0x7f8948100650 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8938009910 tx=0x7f8938009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 -- 172.21.15.37:0/1368415524 shutdown_connections 2024-04-15T22:57:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 --2- 172.21.15.37:0/1368415524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8948101440 0x7f8948101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 --2- 172.21.15.37:0/1368415524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948100240 0x7f8948100650 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 -- 172.21.15.37:0/1368415524 >> 172.21.15.37:0/1368415524 conn(0x7f89480fb7f0 msgr2=0x7f89480fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 -- 172.21.15.37:0/1368415524 shutdown_connections 2024-04-15T22:57:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.288+0000 7f894e760700 1 -- 172.21.15.37:0/1368415524 wait complete. 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f894e760700 1 Processor -- start 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f894e760700 1 -- start start 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f894e760700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8948100240 0x7f8948195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f894e760700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 0x7f8948196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f894e760700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8948196630 con 0x7f8948101440 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f894e760700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8948196770 con 0x7f8948100240 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f893ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 0x7f8948196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f893ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 0x7f8948196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41686/0 (socket says 172.21.15.37:41686) 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f893ffff700 1 -- 172.21.15.37:0/2856648706 learned_addr learned my addr 172.21.15.37:0/2856648706 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.289+0000 7f8947fff700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8948100240 0x7f8948195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f893ffff700 1 -- 172.21.15.37:0/2856648706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8948100240 msgr2=0x7f8948195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:14.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f893ffff700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8948100240 0x7f8948195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f893ffff700 1 -- 172.21.15.37:0/2856648706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380095f0 con 0x7f8948101440 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f893ffff700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 0x7f8948196010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f893000e920 tx=0x7f893000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f893000c9d0 con 0x7f8948101440 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8930010430 con 0x7f8948101440 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894819b220 con 0x7f8948101440 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8930018650 con 0x7f8948101440 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.290+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894819b770 con 0x7f8948101440 2024-04-15T22:57:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.291+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8948108d90 con 0x7f8948101440 2024-04-15T22:57:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.291+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f89300105a0 con 0x7f8948101440 2024-04-15T22:57:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.292+0000 7f8945ffb700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f893406c680 0x7f893406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.292+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f8930014070 con 0x7f8948101440 2024-04-15T22:57:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.292+0000 7f8947fff700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f893406c680 0x7f893406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.293+0000 7f8947fff700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f893406c680 0x7f893406eb30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f893800b6f0 tx=0x7f8938005940 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:14.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.297+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8930051720 con 0x7f8948101440 2024-04-15T22:57:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:14.550+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f894819be00 con 0x7f8948101440 2024-04-15T22:57:14.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:14 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6826/479125410,v1:172.21.15.37:6827/479125410] up:active 2024-04-15T22:57:14.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:14 smithi189 ceph-mon[36179]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 2024-04-15T22:57:14.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:14 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6826/479125410,v1:172.21.15.37:6827/479125410] up:active 2024-04-15T22:57:14.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:14 smithi037 ceph-mon[29191]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 2024-04-15T22:57:15.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.422+0000 7f8945ffb700 1 -- 172.21.15.37:0/2856648706 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8930059de0 con 0x7f8948101440 2024-04-15T22:57:15.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f893406c680 msgr2=0x7f893406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:15.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f893406c680 0x7f893406eb30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f893800b6f0 tx=0x7f8938005940 comp rx=0 tx=0).stop 2024-04-15T22:57:15.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 msgr2=0x7f8948196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:15.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 0x7f8948196010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f893000e920 tx=0x7f893000ec30 comp rx=0 tx=0).stop 2024-04-15T22:57:15.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 shutdown_connections 2024-04-15T22:57:15.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f893406c680 0x7f893406eb30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:15.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8948100240 0x7f8948195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:15.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 --2- 172.21.15.37:0/2856648706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8948101440 0x7f8948196010 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:15.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 >> 172.21.15.37:0/2856648706 conn(0x7f89480fb7f0 msgr2=0x7f8948104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:15.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 shutdown_connections 2024-04-15T22:57:15.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:15.425+0000 7f894e760700 1 -- 172.21.15.37:0/2856648706 wait complete. 2024-04-15T22:57:15.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] up:boot 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 1 up:standby 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.pqjbmb"}]: dispatch 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 1 up:standby 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.ifpdxj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.ifpdxj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: Deploying daemon mds.cephfs.smithi037.ifpdxj on smithi037 2024-04-15T22:57:15.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:15 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2856648706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-15T22:57:15.660 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] up:boot 2024-04-15T22:57:15.660 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 1 up:standby 2024-04-15T22:57:15.660 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.pqjbmb"}]: dispatch 2024-04-15T22:57:15.660 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 1 up:standby 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.ifpdxj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi037.ifpdxj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: Deploying daemon mds.cephfs.smithi037.ifpdxj on smithi037 2024-04-15T22:57:15.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:15 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2856648706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-15T22:57:16.092 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:57:16.102 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:16.102 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-15T22:57:16.564 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:16.676 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-15T22:57:16.676 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2856648706' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-15T22:57:16.676 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: daemon mds.cephfs.smithi189.pqjbmb assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-15T22:57:16.676 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-15T22:57:16.676 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-15T22:57:16.677 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 1 up:standby 2024-04-15T22:57:16.677 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:creating} 2024-04-15T22:57:16.677 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:16 smithi037 ceph-mon[29191]: daemon mds.cephfs.smithi189.pqjbmb is now active in filesystem cephfs as rank 1 2024-04-15T22:57:16.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-15T22:57:16.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2856648706' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-15T22:57:16.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: daemon mds.cephfs.smithi189.pqjbmb assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-15T22:57:16.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-15T22:57:16.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-15T22:57:16.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: fsmap cephfs:1 {0=cephfs.smithi037.rdehay=up:active} 1 up:standby 2024-04-15T22:57:16.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:creating} 2024-04-15T22:57:16.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:16 smithi189 ceph-mon[36179]: daemon mds.cephfs.smithi189.pqjbmb is now active in filesystem cephfs as rank 1 2024-04-15T22:57:17.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:17 smithi037 ceph-mon[29191]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-15T22:57:17.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:17 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] up:active 2024-04-15T22:57:17.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:17 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2024-04-15T22:57:17.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:17 smithi189 ceph-mon[36179]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-15T22:57:17.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:17 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] up:active 2024-04-15T22:57:17.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:17 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2024-04-15T22:57:18.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 -- 172.21.15.37:0/324961592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc100350 msgr2=0x7f09bc1007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 --2- 172.21.15.37:0/324961592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc100350 0x7f09bc1007a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009910 tx=0x7f09b0009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 -- 172.21.15.37:0/324961592 shutdown_connections 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 --2- 172.21.15.37:0/324961592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc100350 0x7f09bc1007a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 --2- 172.21.15.37:0/324961592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09bc0ff150 0x7f09bc0ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 -- 172.21.15.37:0/324961592 >> 172.21.15.37:0/324961592 conn(0x7f09bc0fa700 msgr2=0x7f09bc0fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 -- 172.21.15.37:0/324961592 shutdown_connections 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.209+0000 7f09c46c9700 1 -- 172.21.15.37:0/324961592 wait complete. 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c46c9700 1 Processor -- start 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c46c9700 1 -- start start 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c46c9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 0x7f09bc195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c46c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09bc100350 0x7f09bc195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c46c9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc196600 con 0x7f09bc0ff150 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c46c9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc196740 con 0x7f09bc100350 2024-04-15T22:57:18.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c2465700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 0x7f09bc195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c2465700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 0x7f09bc195aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41702/0 (socket says 172.21.15.37:41702) 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.210+0000 7f09c2465700 1 -- 172.21.15.37:0/4011752245 learned_addr learned my addr 172.21.15.37:0/4011752245 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09c1c64700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09bc100350 0x7f09bc195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09c2465700 1 -- 172.21.15.37:0/4011752245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09bc100350 msgr2=0x7f09bc195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09c2465700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09bc100350 0x7f09bc195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09c2465700 1 -- 172.21.15.37:0/4011752245 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b00095f0 con 0x7f09bc0ff150 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09c2465700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 0x7f09bc195aa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f09b800d900 tx=0x7f09b800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b80041f0 con 0x7f09bc0ff150 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f09b8004350 con 0x7f09bc0ff150 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b8003d20 con 0x7f09bc0ff150 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.211+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc19b1f0 con 0x7f09bc0ff150 2024-04-15T22:57:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.212+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc19b6f0 con 0x7f09bc0ff150 2024-04-15T22:57:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.213+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc107ca0 con 0x7f09bc0ff150 2024-04-15T22:57:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.213+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f09b8009730 con 0x7f09bc0ff150 2024-04-15T22:57:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.214+0000 7f09af7fe700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f09a806c680 0x7f09a806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.214+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f09b808b7a0 con 0x7f09bc0ff150 2024-04-15T22:57:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.214+0000 7f09c1c64700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f09a806c680 0x7f09a806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.214+0000 7f09c1c64700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f09a806c680 0x7f09a806eb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09b0006010 tx=0x7f09b0005a90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.219+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f09b8059910 con 0x7f09bc0ff150 2024-04-15T22:57:18.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:18.470+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f09bc107e60 con 0x7f09bc0ff150 2024-04-15T22:57:19.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.427+0000 7f09af7fe700 1 -- 172.21.15.37:0/4011752245 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09b8003e80 con 0x7f09bc0ff150 2024-04-15T22:57:19.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f09a806c680 msgr2=0x7f09a806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:19.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f09a806c680 0x7f09a806eb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09b0006010 tx=0x7f09b0005a90 comp rx=0 tx=0).stop 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 msgr2=0x7f09bc195aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 0x7f09bc195aa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f09b800d900 tx=0x7f09b800dc10 comp rx=0 tx=0).stop 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 shutdown_connections 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f09a806c680 0x7f09a806eb30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09bc0ff150 0x7f09bc195aa0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 --2- 172.21.15.37:0/4011752245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09bc100350 0x7f09bc195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.429+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 >> 172.21.15.37:0/4011752245 conn(0x7f09bc0fa700 msgr2=0x7f09bc068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.430+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 shutdown_connections 2024-04-15T22:57:19.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:19.430+0000 7f09c46c9700 1 -- 172.21.15.37:0/4011752245 wait complete. 2024-04-15T22:57:19.528 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 5 op/s 2024-04-15T22:57:19.528 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.528 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:19.528 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.528 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/4011752245' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-15T22:57:19.529 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.529 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.529 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.529 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.kzokod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:19.529 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.kzokod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:19.529 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:19 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 5 op/s 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/4011752245' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.kzokod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-15T22:57:19.734 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.kzokod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-15T22:57:19.735 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:19 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:20.196 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:57:20.215 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:20.215 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-15T22:57:20.442 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:20.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: Deploying daemon mds.cephfs.smithi189.kzokod on smithi189 2024-04-15T22:57:20.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-15T22:57:20.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: Cluster is now healthy 2024-04-15T22:57:20.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/4011752245' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:boot 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 1 up:standby 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:20 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi037.ifpdxj"}]: dispatch 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: Deploying daemon mds.cephfs.smithi189.kzokod on smithi189 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: Cluster is now healthy 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/4011752245' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-15T22:57:20.565 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:boot 2024-04-15T22:57:20.566 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 1 up:standby 2024-04-15T22:57:20.566 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:20 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi037.ifpdxj"}]: dispatch 2024-04-15T22:57:21.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:21 smithi189 ceph-mon[36179]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-04-15T22:57:21.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:21 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.682 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:21 smithi037 ceph-mon[29191]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-04-15T22:57:21.682 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.682 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.682 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.682 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.682 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:21 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:21.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.994+0000 7fdd40a84700 1 -- 172.21.15.37:0/2614973087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c1003f0 msgr2=0x7fdd3c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:21.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.994+0000 7fdd40a84700 1 --2- 172.21.15.37:0/2614973087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c1003f0 0x7fdd3c100860 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009910 tx=0x7fdd30009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:21.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 -- 172.21.15.37:0/2614973087 shutdown_connections 2024-04-15T22:57:21.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 --2- 172.21.15.37:0/2614973087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c1003f0 0x7fdd3c100860 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:21.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 --2- 172.21.15.37:0/2614973087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c0ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:21.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 -- 172.21.15.37:0/2614973087 >> 172.21.15.37:0/2614973087 conn(0x7fdd3c0fa7a0 msgr2=0x7fdd3c0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:21.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 -- 172.21.15.37:0/2614973087 shutdown_connections 2024-04-15T22:57:21.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 -- 172.21.15.37:0/2614973087 wait complete. 2024-04-15T22:57:21.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.995+0000 7fdd40a84700 1 Processor -- start 2024-04-15T22:57:21.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd40a84700 1 -- start start 2024-04-15T22:57:21.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd40a84700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:21.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd40a84700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd3c1003f0 0x7fdd3c1017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:21.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd40a84700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c101d10 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd40a84700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c101e50 con 0x7fdd3c1003f0 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd3a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd3a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56506/0 (socket says 172.21.15.37:56506) 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd3a59c700 1 -- 172.21.15.37:0/3532048818 learned_addr learned my addr 172.21.15.37:0/3532048818 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd39d9b700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd3c1003f0 0x7fdd3c1017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd3a59c700 1 -- 172.21.15.37:0/3532048818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd3c1003f0 msgr2=0x7fdd3c1017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd3a59c700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd3c1003f0 0x7fdd3c1017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.996+0000 7fdd3a59c700 1 -- 172.21.15.37:0/3532048818 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd300095f0 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.997+0000 7fdd3a59c700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c103180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00d900 tx=0x7fdd2c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.997+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2c0041f0 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.997+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c102130 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.997+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdd2c004ce0 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.997+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2c00b750 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.997+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c19de20 con 0x7fdd3c0ff150 2024-04-15T22:57:21.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.998+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c068ab0 con 0x7fdd3c0ff150 2024-04-15T22:57:21.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.999+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fdd2c004e50 con 0x7fdd3c0ff150 2024-04-15T22:57:21.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.999+0000 7fdd2b7fe700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdd2406c680 0x7fdd2406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:21.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.999+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fdd2c08b760 con 0x7fdd3c0ff150 2024-04-15T22:57:21.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:21.999+0000 7fdd39d9b700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdd2406c680 0x7fdd2406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:21.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.000+0000 7fdd39d9b700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdd2406c680 0x7fdd2406eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b5c0 tx=0x7fdd30005ee0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:21.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.004+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdd2c0598d0 con 0x7fdd3c0ff150 2024-04-15T22:57:22.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.283+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd3c19dfb0 con 0x7fdd3c0ff150 2024-04-15T22:57:22.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.436+0000 7fdd2b7fe700 1 -- 172.21.15.37:0/3532048818 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v12) v1 ==== 168+0+0 (secure 0 0 0) 0x7fdd2c059460 con 0x7fdd3c0ff150 2024-04-15T22:57:22.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdd2406c680 msgr2=0x7fdd2406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdd2406c680 0x7fdd2406eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b5c0 tx=0x7fdd30005ee0 comp rx=0 tx=0).stop 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 msgr2=0x7fdd3c103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c103180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00d900 tx=0x7fdd2c00dc10 comp rx=0 tx=0).stop 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 shutdown_connections 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdd2406c680 0x7fdd2406eb30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdd3c0ff150 0x7fdd3c103180 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 --2- 172.21.15.37:0/3532048818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd3c1003f0 0x7fdd3c1017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 >> 172.21.15.37:0/3532048818 conn(0x7fdd3c0fa7a0 msgr2=0x7fdd3c068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 shutdown_connections 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:22.438+0000 7fdd40a84700 1 -- 172.21.15.37:0/3532048818 wait complete. 2024-04-15T22:57:22.431 INFO:teuthology.orchestra.run.smithi037.stderr:inline data enabled 2024-04-15T22:57:22.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:22 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.189:6826/1561142822,v1:172.21.15.189:6827/1561142822] up:boot 2024-04-15T22:57:22.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:22 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:22.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:22 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.kzokod"}]: dispatch 2024-04-15T22:57:22.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:22 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:22.462 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:22 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3532048818' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T22:57:22.538 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:22 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.189:6826/1561142822,v1:172.21.15.189:6827/1561142822] up:boot 2024-04-15T22:57:22.538 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:22 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:22.538 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:22 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.kzokod"}]: dispatch 2024-04-15T22:57:22.538 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:22 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:22.538 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:22 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3532048818' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T22:57:23.173 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:57:23.183 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:23.183 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-15T22:57:23.582 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:23.695 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:23 smithi037 ceph-mon[29191]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-15T22:57:23.695 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:23 smithi037 ceph-mon[29191]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-15T22:57:23.695 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:23 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3532048818' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T22:57:23.695 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:23 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:23.695 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:23 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:23.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:23 smithi189 ceph-mon[36179]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-15T22:57:23.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:23 smithi189 ceph-mon[36179]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-15T22:57:23.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:23 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3532048818' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T22:57:23.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:23 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:23.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:23 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:standby 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: Dropping low affinity active daemon mds.cephfs.smithi037.rdehay in favor of higher affinity standby. 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: Replacing daemon mds.cephfs.smithi037.rdehay as rank 0 with standby daemon mds.cephfs.smithi037.ifpdxj 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: osdmap e46: 6 total, 6 up, 6 in 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] up:active 2024-04-15T22:57:24.668 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:replay,1=cephfs.smithi189.pqjbmb=up:active} 1 up:standby 2024-04-15T22:57:24.669 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:24.669 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:24 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:standby 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: Dropping low affinity active daemon mds.cephfs.smithi037.rdehay in favor of higher affinity standby. 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: Replacing daemon mds.cephfs.smithi037.rdehay as rank 0 with standby daemon mds.cephfs.smithi037.ifpdxj 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.rdehay=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: osdmap e46: 6 total, 6 up, 6 in 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] up:active 2024-04-15T22:57:24.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:replay,1=cephfs.smithi189.pqjbmb=up:active} 1 up:standby 2024-04-15T22:57:24.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:24.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:24 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:25.468 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:25 smithi037 ceph-mon[29191]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s wr, 7 op/s 2024-04-15T22:57:25.468 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:25 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:resolve 2024-04-15T22:57:25.468 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:25 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6826/2034364979,v1:172.21.15.37:6827/2034364979] up:boot 2024-04-15T22:57:25.468 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:25 smithi037 ceph-mon[29191]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:resolve,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:25.468 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:25 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi037.rdehay"}]: dispatch 2024-04-15T22:57:25.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:25 smithi189 ceph-mon[36179]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s wr, 7 op/s 2024-04-15T22:57:25.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:25 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:resolve 2024-04-15T22:57:25.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:25 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6826/2034364979,v1:172.21.15.37:6827/2034364979] up:boot 2024-04-15T22:57:25.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:25 smithi189 ceph-mon[36179]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:resolve,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:25.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:25 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi037.rdehay"}]: dispatch 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 -- 172.21.15.37:0/4027445836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 msgr2=0x7fb924100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 --2- 172.21.15.37:0/4027445836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924100840 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb910009910 tx=0x7fb910009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 -- 172.21.15.37:0/4027445836 shutdown_connections 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 --2- 172.21.15.37:0/4027445836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924100840 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 --2- 172.21.15.37:0/4027445836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9240ff130 0x7fb9240ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 -- 172.21.15.37:0/4027445836 >> 172.21.15.37:0/4027445836 conn(0x7fb9240fa780 msgr2=0x7fb9240fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 -- 172.21.15.37:0/4027445836 shutdown_connections 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.906+0000 7fb92abd1700 1 -- 172.21.15.37:0/4027445836 wait complete. 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb92abd1700 1 Processor -- start 2024-04-15T22:57:25.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb92abd1700 1 -- start start 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb92abd1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9240ff130 0x7fb924194960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb92abd1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924194ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb92abd1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241954c0 con 0x7fb9241003d0 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb92abd1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924195600 con 0x7fb9240ff130 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb923fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924194ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb923fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924194ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56528/0 (socket says 172.21.15.37:56528) 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.907+0000 7fb923fff700 1 -- 172.21.15.37:0/2737102414 learned_addr learned my addr 172.21.15.37:0/2737102414 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb92896d700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9240ff130 0x7fb924194960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb923fff700 1 -- 172.21.15.37:0/2737102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9240ff130 msgr2=0x7fb924194960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb923fff700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9240ff130 0x7fb924194960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb923fff700 1 -- 172.21.15.37:0/2737102414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9100095f0 con 0x7fb9241003d0 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb923fff700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924194ea0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb910006010 tx=0x7fb91000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:25.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb91001d070 con 0x7fb9241003d0 2024-04-15T22:57:25.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb91000f460 con 0x7fb9241003d0 2024-04-15T22:57:25.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb924068680 con 0x7fb9241003d0 2024-04-15T22:57:25.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb910021640 con 0x7fb9241003d0 2024-04-15T22:57:25.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.908+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb924068b70 con 0x7fb9241003d0 2024-04-15T22:57:25.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.910+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb924107c10 con 0x7fb9241003d0 2024-04-15T22:57:25.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.911+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb91000fab0 con 0x7fb9241003d0 2024-04-15T22:57:25.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.911+0000 7fb921ffb700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb91406c680 0x7fb91406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:25.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.911+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb91008cb80 con 0x7fb9241003d0 2024-04-15T22:57:25.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.911+0000 7fb92896d700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb91406c680 0x7fb91406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:25.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.912+0000 7fb92896d700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb91406c680 0x7fb91406eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb918005950 tx=0x7fb91800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:25.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:25.916+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb91005ac90 con 0x7fb9241003d0 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.165+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb92419c120 con 0x7fb9241003d0 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.165+0000 7fb921ffb700 1 -- 172.21.15.37:0/2737102414 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fb91005a820 con 0x7fb9241003d0 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout:e16 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout:legacy client fscid: 1 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout:Filesystem 'cephfs' (1) 2024-04-15T22:57:26.158 INFO:teuthology.orchestra.run.smithi037.stdout:fs_name cephfs 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 16 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-04-15T22:57:07.400892+0000 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:modified 2024-04-15T22:57:25.448336+0000 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:tableserver 0 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:root 0 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:session_timeout 60 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:session_autoclose 300 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:max_file_size 1099511627776 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:required_client_features {} 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:last_failure 0 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:last_failure_osd_epoch 46 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:max_mds 2 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:in 0,1 2024-04-15T22:57:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:up {0=14482,1=24289} 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:failed 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:damaged 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:stopped 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:data_pools [3] 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:metadata_pool 2 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:inline_data enabled 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:balancer 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:bal_rank_mask -1 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:standby_count_wanted 1 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.cephfs.smithi037.ifpdxj{0:14482} state up:reconnect seq 4 join_fscid=1 addr [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] compat {c=[1],r=[1],i=[7ff]}] 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.cephfs.smithi189.pqjbmb{1:24289} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.189:6824/2225656114,v1:172.21.15.189:6825/2225656114] compat {c=[1],r=[1],i=[7ff]}] 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:Standby daemons: 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.cephfs.smithi037.rdehay{-1:14492} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.37:6826/2034364979,v1:172.21.15.37:6827/2034364979] compat {c=[1],r=[1],i=[7ff]}] 2024-04-15T22:57:26.161 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.cephfs.smithi189.kzokod{-1:24301} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1561142822,v1:172.21.15.189:6827/1561142822] compat {c=[1],r=[1],i=[7ff]}] 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.169+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb91406c680 msgr2=0x7fb91406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.169+0000 7fb92abd1700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb91406c680 0x7fb91406eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb918005950 tx=0x7fb91800b410 comp rx=0 tx=0).stop 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.169+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 msgr2=0x7fb924194ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.169+0000 7fb92abd1700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924194ea0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb910006010 tx=0x7fb91000bd70 comp rx=0 tx=0).stop 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 shutdown_connections 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb91406c680 0x7fb91406eb30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9240ff130 0x7fb924194960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 --2- 172.21.15.37:0/2737102414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9241003d0 0x7fb924194ea0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 >> 172.21.15.37:0/2737102414 conn(0x7fb9240fa780 msgr2=0x7fb924103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 shutdown_connections 2024-04-15T22:57:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:26.170+0000 7fb92abd1700 1 -- 172.21.15.37:0/2737102414 wait complete. 2024-04-15T22:57:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 16 2024-04-15T22:57:26.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:26 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:reconnect 2024-04-15T22:57:26.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:26 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.189:6826/1561142822,v1:172.21.15.189:6827/1561142822] up:standby 2024-04-15T22:57:26.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:26 smithi189 ceph-mon[36179]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:reconnect,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:26.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:26 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2737102414' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-15T22:57:26.724 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:26 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:reconnect 2024-04-15T22:57:26.724 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:26 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.189:6826/1561142822,v1:172.21.15.189:6827/1561142822] up:standby 2024-04-15T22:57:26.724 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:26 smithi037 ceph-mon[29191]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:reconnect,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:26.724 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:26 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2737102414' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-15T22:57:27.026 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-15T22:57:27.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:27 smithi037 ceph-mon[29191]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-15T22:57:27.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:27 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:rejoin 2024-04-15T22:57:27.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:27 smithi037 ceph-mon[29191]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:rejoin,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:27.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:27 smithi037 ceph-mon[29191]: daemon mds.cephfs.smithi037.ifpdxj is now active in filesystem cephfs as rank 0 2024-04-15T22:57:27.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:27 smithi189 ceph-mon[36179]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-15T22:57:27.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:27 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:rejoin 2024-04-15T22:57:27.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:27 smithi189 ceph-mon[36179]: fsmap cephfs:2/2 {0=cephfs.smithi037.ifpdxj=up:rejoin,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:27.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:27 smithi189 ceph-mon[36179]: daemon mds.cephfs.smithi037.ifpdxj is now active in filesystem cephfs as rank 0 2024-04-15T22:57:27.820 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:28.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:28 smithi037 ceph-mon[29191]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-15T22:57:28.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:28 smithi037 ceph-mon[29191]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:active 2024-04-15T22:57:28.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:28 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.ifpdxj=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:28.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:28 smithi189 ceph-mon[36179]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-15T22:57:28.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:28 smithi189 ceph-mon[36179]: mds.? [v2:172.21.15.37:6828/2461231902,v1:172.21.15.37:6829/2461231902] up:active 2024-04-15T22:57:28.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:28 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.ifpdxj=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:57:29.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:29 smithi037 ceph-mon[29191]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-15T22:57:29.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:29 smithi189 ceph-mon[36179]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-15T22:57:30.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 -- 172.21.15.37:0/3803500624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fe500 msgr2=0x7f55d00fe910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 --2- 172.21.15.37:0/3803500624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fe500 0x7f55d00fe910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009910 tx=0x7f55c0009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 -- 172.21.15.37:0/3803500624 shutdown_connections 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 --2- 172.21.15.37:0/3803500624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fee50 0x7f55d0101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 --2- 172.21.15.37:0/3803500624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fe500 0x7f55d00fe910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 -- 172.21.15.37:0/3803500624 >> 172.21.15.37:0/3803500624 conn(0x7f55d00f9f10 msgr2=0x7f55d00fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 -- 172.21.15.37:0/3803500624 shutdown_connections 2024-04-15T22:57:30.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.048+0000 7f55d57e2700 1 -- 172.21.15.37:0/3803500624 wait complete. 2024-04-15T22:57:30.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.049+0000 7f55d57e2700 1 Processor -- start 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.049+0000 7f55d57e2700 1 -- start start 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.049+0000 7f55d57e2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 0x7f55d0195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.049+0000 7f55d57e2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 0x7f55d0196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.049+0000 7f55d57e2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d0196660 con 0x7f55d00fee50 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.049+0000 7f55d57e2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d01967a0 con 0x7f55d00fe500 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 0x7f55d0196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 0x7f55d0195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 0x7f55d0196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58732/0 (socket says 172.21.15.37:58732) 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 0x7f55d0195b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.37:38442/0 (socket says 172.21.15.37:38442) 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 -- 172.21.15.37:0/2528216775 learned_addr learned my addr 172.21.15.37:0/2528216775 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 -- 172.21.15.37:0/2528216775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 msgr2=0x7f55d0195b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 0x7f55d0195b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 -- 172.21.15.37:0/2528216775 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c00095f0 con 0x7f55d00fee50 2024-04-15T22:57:30.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ceffd700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 0x7f55d0195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.050+0000 7f55ce7fc700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 0x7f55d0196040 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f55c400be80 tx=0x7f55c400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.051+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c4012c60 con 0x7f55d00fee50 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.051+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f55c4016070 con 0x7f55d00fee50 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.051+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d00ff3d0 con 0x7f55d00fee50 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.051+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d00ff920 con 0x7f55d00fee50 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.051+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c401b6e0 con 0x7f55d00fee50 2024-04-15T22:57:30.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.052+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc005320 con 0x7f55d00fee50 2024-04-15T22:57:30.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.053+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f55c401c400 con 0x7f55d00fee50 2024-04-15T22:57:30.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.053+0000 7f55b7fff700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f55b806c680 0x7f55b806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:30.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.054+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f55c408f200 con 0x7f55d00fee50 2024-04-15T22:57:30.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.054+0000 7f55ceffd700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f55b806c680 0x7f55b806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:30.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.054+0000 7f55ceffd700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f55b806c680 0x7f55b806eb30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f55c0006010 tx=0x7f55c000b540 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:30.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.060+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f55c4055720 con 0x7f55d00fee50 2024-04-15T22:57:30.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.323+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55bc005f70 con 0x7f55d00fee50 2024-04-15T22:57:30.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.324+0000 7f55b7fff700 1 -- 172.21.15.37:0/2528216775 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4760 (secure 0 0 0) 0x7f55c405d390 con 0x7f55d00fee50 2024-04-15T22:57:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f55b806c680 msgr2=0x7f55b806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f55b806c680 0x7f55b806eb30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f55c0006010 tx=0x7f55c000b540 comp rx=0 tx=0).stop 2024-04-15T22:57:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 msgr2=0x7f55d0196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 0x7f55d0196040 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f55c400be80 tx=0x7f55c400dbb0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 shutdown_connections 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f55b806c680 0x7f55b806eb30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d00fe500 0x7f55d0195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 --2- 172.21.15.37:0/2528216775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55d00fee50 0x7f55d0196040 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 >> 172.21.15.37:0/2528216775 conn(0x7f55d00f9f10 msgr2=0x7f55d01047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 shutdown_connections 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:30.326+0000 7f55d57e2700 1 -- 172.21.15.37:0/2528216775 wait complete. 2024-04-15T22:57:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 18 2024-04-15T22:57:30.328 INFO:teuthology.orchestra.run.smithi037.stdout:true 2024-04-15T22:57:30.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:30.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:30.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:30.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:30.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:30.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:30.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2528216775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-15T22:57:30.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:30 smithi189 ceph-mon[36179]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-15T22:57:30.859 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:30.859 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:30.859 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:30.859 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:30.859 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:30.859 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:30.860 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2528216775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-15T22:57:30.860 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:30 smithi037 ceph-mon[29191]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-15T22:57:31.479 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-15T22:57:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:33.065 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:33.065 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:32 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:33.065 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:32 smithi037 ceph-mon[29191]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-15T22:57:33.212 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:33.212 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:32 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:33.212 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:32 smithi189 ceph-mon[36179]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 -- 172.21.15.37:0/2957816500 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 msgr2=0x7fa8580fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/2957816500 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580fe300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa848009910 tx=0x7fa848009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 -- 172.21.15.37:0/2957816500 shutdown_connections 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/2957816500 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580fe300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/2957816500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa858101880 0x7fa858101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 -- 172.21.15.37:0/2957816500 >> 172.21.15.37:0/2957816500 conn(0x7fa8580f9f10 msgr2=0x7fa8580fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 -- 172.21.15.37:0/2957816500 shutdown_connections 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 -- 172.21.15.37:0/2957816500 wait complete. 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.942+0000 7fa85fcc5700 1 Processor -- start 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85fcc5700 1 -- start start 2024-04-15T22:57:33.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85fcc5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa858101880 0x7fa858100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85fcc5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580ff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85fcc5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8580ffb10 con 0x7fa8581021d0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85fcc5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8580ffc50 con 0x7fa858101880 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580ff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580ff5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58752/0 (socket says 172.21.15.37:58752) 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 -- 172.21.15.37:0/353533696 learned_addr learned my addr 172.21.15.37:0/353533696 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85da61700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa858101880 0x7fa858100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 -- 172.21.15.37:0/353533696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa858101880 msgr2=0x7fa858100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa858101880 0x7fa858100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 -- 172.21.15.37:0/353533696 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8480095f0 con 0x7fa8581021d0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa85d260700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580ff5d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa84800bda0 tx=0x7fa84800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.943+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa84801d070 con 0x7fa8581021d0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.944+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8580ffed0 con 0x7fa8581021d0 2024-04-15T22:57:33.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.944+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa84800f460 con 0x7fa8581021d0 2024-04-15T22:57:33.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.944+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa848021640 con 0x7fa8581021d0 2024-04-15T22:57:33.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.944+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8581003c0 con 0x7fa8581021d0 2024-04-15T22:57:33.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.945+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa858108d20 con 0x7fa8581021d0 2024-04-15T22:57:33.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.946+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa84800fa00 con 0x7fa8581021d0 2024-04-15T22:57:33.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.946+0000 7fa84effd700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa84406c680 0x7fa84406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:33.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.946+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa84808cbd0 con 0x7fa8581021d0 2024-04-15T22:57:33.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.946+0000 7fa85da61700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa84406c680 0x7fa84406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:33.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.947+0000 7fa85da61700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa84406c680 0x7fa84406eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa854005950 tx=0x7fa8540058e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:33.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:33.951+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa84805ace0 con 0x7fa8581021d0 2024-04-15T22:57:34.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.258+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fa858108f10 con 0x7fa8581021d0 2024-04-15T22:57:34.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.258+0000 7fa84effd700 1 -- 172.21.15.37:0/353533696 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+83 (secure 0 0 0) 0x7fa84805a870 con 0x7fa8581021d0 2024-04-15T22:57:34.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa84406c680 msgr2=0x7fa84406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:34.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa84406c680 0x7fa84406eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa854005950 tx=0x7fa8540058e0 comp rx=0 tx=0).stop 2024-04-15T22:57:34.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 msgr2=0x7fa8580ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:34.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580ff5d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa84800bda0 tx=0x7fa84800be80 comp rx=0 tx=0).stop 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 shutdown_connections 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa84406c680 0x7fa84406eb30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa858101880 0x7fa858100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 --2- 172.21.15.37:0/353533696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8581021d0 0x7fa8580ff5d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 >> 172.21.15.37:0/353533696 conn(0x7fa8580f9f10 msgr2=0x7fa858104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 shutdown_connections 2024-04-15T22:57:34.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:34.260+0000 7fa85fcc5700 1 -- 172.21.15.37:0/353533696 wait complete. 2024-04-15T22:57:34.262 INFO:teuthology.orchestra.run.smithi037.stdout:true 2024-04-15T22:57:34.500 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:34.500 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:34.500 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:34 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/353533696' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-15T22:57:34.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:34.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:34.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:34 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/353533696' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-15T22:57:35.187 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-15T22:57:35.200 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T22:57:35.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 -- 172.21.15.37:0/2487026669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c100850 msgr2=0x7f850c100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 --2- 172.21.15.37:0/2487026669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c100850 0x7f850c100cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009880 tx=0x7f84f402bce0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 -- 172.21.15.37:0/2487026669 shutdown_connections 2024-04-15T22:57:35.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 --2- 172.21.15.37:0/2487026669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c100850 0x7f850c100cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 --2- 172.21.15.37:0/2487026669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f850c0ff660 0x7f850c0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 -- 172.21.15.37:0/2487026669 >> 172.21.15.37:0/2487026669 conn(0x7f850c0fabc0 msgr2=0x7f850c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 -- 172.21.15.37:0/2487026669 shutdown_connections 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.351+0000 7f851166a640 1 -- 172.21.15.37:0/2487026669 wait complete. 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f851166a640 1 Processor -- start 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f851166a640 1 -- start start 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f851166a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 0x7f850c074010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f851166a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f850c100850 0x7f850c072650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f851166a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c072b90 con 0x7f850c0ff660 2024-04-15T22:57:35.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f851166a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c072d00 con 0x7f850c100850 2024-04-15T22:57:35.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f850affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 0x7f850c074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f850affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 0x7f850c074010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58774/0 (socket says 172.21.15.37:58774) 2024-04-15T22:57:35.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.352+0000 7f850affd640 1 -- 172.21.15.37:0/56209562 learned_addr learned my addr 172.21.15.37:0/56209562 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:35.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f85027fc640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f850c100850 0x7f850c072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f850affd640 1 -- 172.21.15.37:0/56209562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f850c100850 msgr2=0x7f850c072650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f850affd640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f850c100850 0x7f850c072650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f850affd640 1 -- 172.21.15.37:0/56209562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f4009560 con 0x7f850c0ff660 2024-04-15T22:57:35.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f850affd640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 0x7f850c074010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00d8d0 tx=0x7f84fc00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:35.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84fc00bea0 con 0x7f850c0ff660 2024-04-15T22:57:35.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c072fe0 con 0x7f850c0ff660 2024-04-15T22:57:35.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f84fc009c00 con 0x7f850c0ff660 2024-04-15T22:57:35.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84fc00f780 con 0x7f850c0ff660 2024-04-15T22:57:35.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.353+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c19cf10 con 0x7f850c0ff660 2024-04-15T22:57:35.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.354+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c108230 con 0x7f850c0ff660 2024-04-15T22:57:35.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.355+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f84fc00fa40 con 0x7f850c0ff660 2024-04-15T22:57:35.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.355+0000 7f8508ff9640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f84f806c6b0 0x7f84f806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.355+0000 7f85027fc640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f84f806c6b0 0x7f84f806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.355+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f84fc08b930 con 0x7f850c0ff660 2024-04-15T22:57:35.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.356+0000 7f85027fc640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f84f806c6b0 0x7f84f806eb60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f850c073d80 tx=0x7f84f4000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:35.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.360+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f84fc055fb0 con 0x7f850c0ff660 2024-04-15T22:57:35.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.535+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f850c1084b0 con 0x7f850c0ff660 2024-04-15T22:57:35.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.536+0000 7f8508ff9640 1 -- 172.21.15.37:0/56209562 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15106 (secure 0 0 0) 0x7f84fc0595d0 con 0x7f850c0ff660 2024-04-15T22:57:35.528 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:35.528 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":46,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","created":"2024-04-15T22:49:47.691565+0000","modified":"2024-04-15T22:57:23.443638+0000","last_up_change":"2024-04-15T22:55:51.865136+0000","last_in_change":"2024-04-15T22:55:32.171674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:54:34.664768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-15T22:57:06.044854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-15T22:57:06.646368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc1abff5-8aea-47ca-982a-29cd590fc31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6803","nonce":2814326386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6805","nonce":2814326386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6809","nonce":2814326386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6807","nonce":2814326386}]},"public_addr":"172.21.15.37:6803/2814326386","cluster_addr":"172.21.15.37:6805/2814326386","heartbeat_back_addr":"172.21.15.37:6809/2814326386","heartbeat_front_addr":"172.21.15.37:6807/2814326386","state":["exists","up"]},{"osd":1,"uuid":"e875b0b4-8220-4f8a-9d7b-c2dffdd988dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6811","nonce":1887622200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6813","nonce":1887622200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6817","nonce":1887622200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6815","nonce":1887622200}]},"public_addr":"172.21.15.37:6811/1887622200","cluster_addr":"172.21.15.37:6813/1887622200","heartbeat_back_addr":"172.21.15.37:6817/1887622200","heartbeat_front_addr":"172.21.15.37:6815/1887622200","state":["exists","up"]},{"osd":2,"uuid":"08257888-e768-4845-9efa-8358f6e8caec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6819","nonce":503233976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6821","nonce":503233976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6825","nonce":503233976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6823","nonce":503233976}]},"public_addr":"172.21.15.37:6819/503233976","cluster_addr":"172.21.15.37:6821/503233976","heartbeat_back_addr":"172.21.15.37:6825/503233976","heartbeat_front_addr":"172.21.15.37:6823/503233976","state":["exists","up"]},{"osd":3,"uuid":"1d49e4a2-0131-436b-b02c-77dd4237bee8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6801","nonce":1840940998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6803","nonce":1840940998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6807","nonce":1840940998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6805","nonce":1840940998}]},"public_addr":"172.21.15.189:6801/1840940998","cluster_addr":"172.21.15.189:6803/1840940998","heartbeat_back_addr":"172.21.15.189:6807/1840940998","heartbeat_front_addr":"172.21.15.189:6805/1840940998","state":["exists","up"]},{"osd":4,"uuid":"b93aee26-d0dd-4cbc-b45e-b005033c1a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6809","nonce":2518250570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6811","nonce":2518250570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6815","nonce":2518250570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6813","nonce":2518250570}]},"public_addr":"172.21.15.189:6809/2518250570","cluster_addr":"172.21.15.189:6811/2518250570","heartbeat_back_addr":"172.21.15.189:6815/2518250570","heartbeat_front_addr":"172.21.15.189:6813/2518250570","state":["exists","up"]},{"osd":5,"uuid":"de638b57-8f16-4a8f-b0b0-bbc7d84831c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6817","nonce":580752685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6819","nonce":580752685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6823","nonce":580752685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6821","nonce":580752685}]},"public_addr":"172.21.15.189:6817/580752685","cluster_addr":"172.21.15.189:6819/580752685","heartbeat_back_addr":"172.21.15.189:6823/580752685","heartbeat_front_addr":"172.21.15.189:6821/580752685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:53:35.515515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:02.780098+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:31.718613+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:56.852313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:23.477486+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:50.019389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3522001690":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/1091007617":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3560860596":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6801/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/957260578":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6800/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/8957815":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/2768618180":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/3190201420":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3357942121":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6827/479125410":"2024-04-16T22:57:23.443615+0000","172.21.15.37:6800/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/1434563686":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6826/479125410":"2024-04-16T22:57:23.443615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:57:35.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f84f806c6b0 msgr2=0x7f84f806eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f84f806c6b0 0x7f84f806eb60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f850c073d80 tx=0x7f84f4000bc0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 msgr2=0x7f850c074010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 0x7f850c074010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00d8d0 tx=0x7f84fc00dd90 comp rx=0 tx=0).stop 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 shutdown_connections 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f84f806c6b0 0x7f84f806eb60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f850c100850 0x7f850c072650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 --2- 172.21.15.37:0/56209562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f850c0ff660 0x7f850c074010 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 >> 172.21.15.37:0/56209562 conn(0x7f850c0fabc0 msgr2=0x7f850c103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 shutdown_connections 2024-04-15T22:57:35.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.538+0000 7f851166a640 1 -- 172.21.15.37:0/56209562 wait complete. 2024-04-15T22:57:35.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T22:54:34.664768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-15T22:57:06.044854+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-15T22:57:06.646368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T22:57:35.542 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T22:57:35.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 -- 172.21.15.37:0/1910116917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd364100870 msgr2=0x7fd364100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 --2- 172.21.15.37:0/1910116917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd364100870 0x7fd364100ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009880 tx=0x7fd34c02bce0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 -- 172.21.15.37:0/1910116917 shutdown_connections 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 --2- 172.21.15.37:0/1910116917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd364100870 0x7fd364100ce0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 --2- 172.21.15.37:0/1910116917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3640ff5e0 0x7fd3640ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 -- 172.21.15.37:0/1910116917 >> 172.21.15.37:0/1910116917 conn(0x7fd3640fac20 msgr2=0x7fd3640fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 -- 172.21.15.37:0/1910116917 shutdown_connections 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.663+0000 7fd36a718640 1 -- 172.21.15.37:0/1910116917 wait complete. 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd36a718640 1 Processor -- start 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd36a718640 1 -- start start 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd36a718640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 0x7fd364194f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd36a718640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364100870 0x7fd364195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd36a718640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd364195a30 con 0x7fd3640ff5e0 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd36a718640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd364195ba0 con 0x7fd364100870 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd363fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 0x7fd364194f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd363fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 0x7fd364194f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58802/0 (socket says 172.21.15.37:58802) 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd363fff640 1 -- 172.21.15.37:0/1978893370 learned_addr learned my addr 172.21.15.37:0/1978893370 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd35bfff640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364100870 0x7fd364195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd363fff640 1 -- 172.21.15.37:0/1978893370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364100870 msgr2=0x7fd364195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd363fff640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364100870 0x7fd364195460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.664+0000 7fd363fff640 1 -- 172.21.15.37:0/1978893370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c009560 con 0x7fd3640ff5e0 2024-04-15T22:57:35.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.665+0000 7fd363fff640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 0x7fd364194f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd35400e970 tx=0x7fd35400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:35.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.665+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd35400cc90 con 0x7fd3640ff5e0 2024-04-15T22:57:35.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.665+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd35400cdf0 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.665+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd354010430 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.665+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3640724f0 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.665+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd364072a50 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.667+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3641082d0 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.667+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fd3540046c0 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.668+0000 7fd361ffb640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fd350070a50 0x7fd350072f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.668+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd354014070 con 0x7fd3640ff5e0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.668+0000 7fd35bfff640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fd350070a50 0x7fd350072f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.668+0000 7fd35bfff640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fd350070a50 0x7fd350072f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd34c02c7c0 tx=0x7fd34c005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:35.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.672+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd354056580 con 0x7fd3640ff5e0 2024-04-15T22:57:35.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:35 smithi189 ceph-mon[36179]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-15T22:57:35.721 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:35 smithi037 ceph-mon[29191]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-15T22:57:35.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.864+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd364073200 con 0x7fd3640ff5e0 2024-04-15T22:57:35.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.865+0000 7fd361ffb640 1 -- 172.21.15.37:0/1978893370 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd354059ba0 con 0x7fd3640ff5e0 2024-04-15T22:57:35.857 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fd350070a50 msgr2=0x7fd350072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fd350070a50 0x7fd350072f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd34c02c7c0 tx=0x7fd34c005fd0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 msgr2=0x7fd364194f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 0x7fd364194f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd35400e970 tx=0x7fd35400ed70 comp rx=0 tx=0).stop 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 shutdown_connections 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fd350070a50 0x7fd350072f00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364100870 0x7fd364195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.866+0000 7fd36a718640 1 --2- 172.21.15.37:0/1978893370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3640ff5e0 0x7fd364194f20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.867+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 >> 172.21.15.37:0/1978893370 conn(0x7fd3640fac20 msgr2=0x7fd364103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.867+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 shutdown_connections 2024-04-15T22:57:35.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.867+0000 7fd36a718640 1 -- 172.21.15.37:0/1978893370 wait complete. 2024-04-15T22:57:35.869 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 -- 172.21.15.37:0/1204065316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2618100840 msgr2=0x7f2618100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 --2- 172.21.15.37:0/1204065316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2618100840 0x7f2618100cb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f260c009880 tx=0x7f260c02bce0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 -- 172.21.15.37:0/1204065316 shutdown_connections 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 --2- 172.21.15.37:0/1204065316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2618100840 0x7f2618100cb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 --2- 172.21.15.37:0/1204065316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26180ff650 0x7f26180ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 -- 172.21.15.37:0/1204065316 >> 172.21.15.37:0/1204065316 conn(0x7f26180fabd0 msgr2=0x7f26180fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 -- 172.21.15.37:0/1204065316 shutdown_connections 2024-04-15T22:57:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.986+0000 7f261dfb1640 1 -- 172.21.15.37:0/1204065316 wait complete. 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f261dfb1640 1 Processor -- start 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f261dfb1640 1 -- start start 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f261dfb1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 0x7f2618194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f261dfb1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2618100840 0x7f2618195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f261dfb1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2618195a60 con 0x7f26180ff650 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f261dfb1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2618195bd0 con 0x7f2618100840 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f26177fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 0x7f2618194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f26177fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 0x7f2618194f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58808/0 (socket says 172.21.15.37:58808) 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f26177fe640 1 -- 172.21.15.37:0/466380428 learned_addr learned my addr 172.21.15.37:0/466380428 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f2616ffd640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2618100840 0x7f2618195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f26177fe640 1 -- 172.21.15.37:0/466380428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2618100840 msgr2=0x7f2618195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f26177fe640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2618100840 0x7f2618195490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.987+0000 7f26177fe640 1 -- 172.21.15.37:0/466380428 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c009560 con 0x7f26180ff650 2024-04-15T22:57:35.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.988+0000 7f26177fe640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 0x7f2618194f50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f260800d8d0 tx=0x7f260800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:35.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.988+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2608009a30 con 0x7f26180ff650 2024-04-15T22:57:35.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.988+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261819a660 con 0x7f26180ff650 2024-04-15T22:57:35.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.988+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2608009b90 con 0x7f26180ff650 2024-04-15T22:57:35.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.988+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f260800f680 con 0x7f26180ff650 2024-04-15T22:57:35.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.988+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261819abc0 con 0x7f26180ff650 2024-04-15T22:57:35.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.989+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2618108390 con 0x7f26180ff650 2024-04-15T22:57:35.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.990+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2608009d00 con 0x7f26180ff650 2024-04-15T22:57:35.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.990+0000 7f2614ff9640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f260006c5f0 0x7f260006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:35.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.990+0000 7f2616ffd640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f260006c5f0 0x7f260006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:35.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.990+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f260808b740 con 0x7f26180ff650 2024-04-15T22:57:35.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.991+0000 7f2616ffd640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f260006c5f0 0x7f260006eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2618196460 tx=0x7f260c000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:35.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:35.995+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2608055dc0 con 0x7f26180ff650 2024-04-15T22:57:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.177+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f261819ade0 con 0x7f26180ff650 2024-04-15T22:57:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.178+0000 7f2614ff9640 1 -- 172.21.15.37:0/466380428 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f26080593e0 con 0x7f26180ff650 2024-04-15T22:57:36.171 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f260006c5f0 msgr2=0x7f260006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f260006c5f0 0x7f260006eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2618196460 tx=0x7f260c000bc0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 msgr2=0x7f2618194f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 0x7f2618194f50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f260800d8d0 tx=0x7f260800dd90 comp rx=0 tx=0).stop 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 shutdown_connections 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f260006c5f0 0x7f260006eaa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2618100840 0x7f2618195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 --2- 172.21.15.37:0/466380428 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26180ff650 0x7f2618194f50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 >> 172.21.15.37:0/466380428 conn(0x7f26180fabd0 msgr2=0x7f2618103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 shutdown_connections 2024-04-15T22:57:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.180+0000 7f261dfb1640 1 -- 172.21.15.37:0/466380428 wait complete. 2024-04-15T22:57:36.183 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2028450364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0fe570 msgr2=0x7fe8dc0fe980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2028450364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0fe570 0x7fe8dc0fe980 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009880 tx=0x7fe8cc02bce0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2028450364 shutdown_connections 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2028450364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2028450364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0fe570 0x7fe8dc0fe980 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2028450364 >> 172.21.15.37:0/2028450364 conn(0x7fe8dc0f9ad0 msgr2=0x7fe8dc0fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2028450364 shutdown_connections 2024-04-15T22:57:36.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.299+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2028450364 wait complete. 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8e29a4640 1 Processor -- start 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8e29a4640 1 -- start start 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8e29a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8dc0fe570 0x7fe8dc068f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8e29a4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8e29a4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc19a2b0 con 0x7fe8dc0ff760 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8e29a4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc19a3f0 con 0x7fe8dc0fe570 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8d3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8d3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58834/0 (socket says 172.21.15.37:58834) 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8d3fff640 1 -- 172.21.15.37:0/2343435323 learned_addr learned my addr 172.21.15.37:0/2343435323 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.300+0000 7fe8dbfff640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8dc0fe570 0x7fe8dc068f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d3fff640 1 -- 172.21.15.37:0/2343435323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8dc0fe570 msgr2=0x7fe8dc068f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d3fff640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8dc0fe570 0x7fe8dc068f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d3fff640 1 -- 172.21.15.37:0/2343435323 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc009560 con 0x7fe8dc0ff760 2024-04-15T22:57:36.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d3fff640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc199ce0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe8c400c810 tx=0x7fe8c400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:36.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c40041f0 con 0x7fe8dc0ff760 2024-04-15T22:57:36.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe8c4007e60 con 0x7fe8dc0ff760 2024-04-15T22:57:36.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc19a6a0 con 0x7fe8dc0ff760 2024-04-15T22:57:36.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c40049d0 con 0x7fe8dc0ff760 2024-04-15T22:57:36.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.301+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc19ac00 con 0x7fe8dc0ff760 2024-04-15T22:57:36.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.302+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc107280 con 0x7fe8dc0ff760 2024-04-15T22:57:36.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.302+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe8c40079b0 con 0x7fe8dc0ff760 2024-04-15T22:57:36.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.303+0000 7fe8d9ffb640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe8c806c5f0 0x7fe8c806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:36.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.303+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe8c408b6e0 con 0x7fe8dc0ff760 2024-04-15T22:57:36.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.303+0000 7fe8dbfff640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe8c806c5f0 0x7fe8c806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:36.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.303+0000 7fe8dbfff640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe8c806c5f0 0x7fe8c806eaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006010 tx=0x7fe8cc000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:36.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.307+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe8c4055d60 con 0x7fe8dc0ff760 2024-04-15T22:57:36.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.497+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fe8dc107410 con 0x7fe8dc0ff760 2024-04-15T22:57:36.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.497+0000 7fe8d9ffb640 1 -- 172.21.15.37:0/2343435323 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe8dc107410 con 0x7fe8dc0ff760 2024-04-15T22:57:36.489 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-04-15T22:57:36.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe8c806c5f0 msgr2=0x7fe8c806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe8c806c5f0 0x7fe8c806eaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006010 tx=0x7fe8cc000bc0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 msgr2=0x7fe8dc199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc199ce0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe8c400c810 tx=0x7fe8c400ccd0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 shutdown_connections 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe8c806c5f0 0x7fe8c806eaa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8dc0ff760 0x7fe8dc199ce0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 --2- 172.21.15.37:0/2343435323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8dc0fe570 0x7fe8dc068f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 >> 172.21.15.37:0/2343435323 conn(0x7fe8dc0f9ad0 msgr2=0x7fe8dc102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 shutdown_connections 2024-04-15T22:57:36.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.499+0000 7fe8e29a4640 1 -- 172.21.15.37:0/2343435323 wait complete. 2024-04-15T22:57:36.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T22:57:36.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/3644546672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740fe540 msgr2=0x7fcf740fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/3644546672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740fe540 0x7fcf740fe950 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcf68009880 tx=0x7fcf6802bce0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/3644546672 shutdown_connections 2024-04-15T22:57:36.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/3644546672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740ff730 0x7fcf740ffba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/3644546672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740fe540 0x7fcf740fe950 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/3644546672 >> 172.21.15.37:0/3644546672 conn(0x7fcf740f9ac0 msgr2=0x7fcf740fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/3644546672 shutdown_connections 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/3644546672 wait complete. 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 Processor -- start 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- start start 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740fe540 0x7fcf74068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 0x7fcf74199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7419a230 con 0x7fcf740ff730 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.616+0000 7fcf7b1a4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7419a370 con 0x7fcf740fe540 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 0x7fcf74199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 0x7fcf74199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58840/0 (socket says 172.21.15.37:58840) 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 -- 172.21.15.37:0/583780950 learned_addr learned my addr 172.21.15.37:0/583780950 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf78f40640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740fe540 0x7fcf74068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 -- 172.21.15.37:0/583780950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740fe540 msgr2=0x7fcf74068f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740fe540 0x7fcf74068f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 -- 172.21.15.37:0/583780950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf68009560 con 0x7fcf740ff730 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf78f40640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740fe540 0x7fcf74068f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf73fff640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 0x7fcf74199c60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000b420 tx=0x7fcf6000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf60011800 con 0x7fcf740ff730 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcf60011e10 con 0x7fcf740ff730 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7419a650 con 0x7fcf740ff730 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf6000f2e0 con 0x7fcf740ff730 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.617+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7419ac30 con 0x7fcf740ff730 2024-04-15T22:57:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.618+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf741072d0 con 0x7fcf740ff730 2024-04-15T22:57:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.619+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fcf60010380 con 0x7fcf740ff730 2024-04-15T22:57:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.619+0000 7fcf71ffb640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fcf6406c6b0 0x7fcf6406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.619+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcf6008b2f0 con 0x7fcf740ff730 2024-04-15T22:57:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.619+0000 7fcf78f40640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fcf6406c6b0 0x7fcf6406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.620+0000 7fcf78f40640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fcf6406c6b0 0x7fcf6406eb60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcf68036040 tx=0x7fcf68005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:36.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.624+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcf60055970 con 0x7fcf740ff730 2024-04-15T22:57:36.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:36 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/56209562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:57:36.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:36 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1978893370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:57:36.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:36 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/466380428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-15T22:57:36.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:36 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/56209562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:57:36.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:36 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1978893370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:57:36.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:36 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/466380428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-15T22:57:36.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.838+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf7419b350 con 0x7fcf740ff730 2024-04-15T22:57:36.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.839+0000 7fcf71ffb640 1 -- 172.21.15.37:0/583780950 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4760 (secure 0 0 0) 0x7fcf60058f90 con 0x7fcf740ff730 2024-04-15T22:57:36.831 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:57:36.831 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14492,"name":"cephfs.smithi037.rdehay","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6827/2034364979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":2034364979},{"type":"v1","addr":"172.21.15.37:6827","nonce":2034364979}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24301,"name":"cephfs.smithi189.kzokod","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6827/1561142822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":1561142822},{"type":"v1","addr":"172.21.15.189:6827","nonce":1561142822}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T22:57:07.400892+0000","modified":"2024-04-15T22:57:27.460254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24289},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi037.ifpdxj","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.37:6829/2461231902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":2461231902},{"type":"v1","addr":"172.21.15.37:6829","nonce":2461231902}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24289":{"gid":24289,"name":"cephfs.smithi189.pqjbmb","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6825/2225656114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2225656114},{"type":"v1","addr":"172.21.15.189:6825","nonce":2225656114}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-15T22:57:36.831 INFO:teuthology.orchestra.run.smithi037.stderr:dumped fsmap epoch 18 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fcf6406c6b0 msgr2=0x7fcf6406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fcf6406c6b0 0x7fcf6406eb60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcf68036040 tx=0x7fcf68005fd0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 msgr2=0x7fcf74199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 0x7fcf74199c60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000b420 tx=0x7fcf6000b8e0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 shutdown_connections 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fcf6406c6b0 0x7fcf6406eb60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf740ff730 0x7fcf74199c60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 --2- 172.21.15.37:0/583780950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf740fe540 0x7fcf74068f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 >> 172.21.15.37:0/583780950 conn(0x7fcf740f9ac0 msgr2=0x7fcf74102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:36.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 shutdown_connections 2024-04-15T22:57:36.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:36.842+0000 7fcf7b1a4640 1 -- 172.21.15.37:0/583780950 wait complete. 2024-04-15T22:57:36.846 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 18} 2024-04-15T22:57:36.846 INFO:teuthology.run_tasks:Running task kclient... 2024-04-15T22:57:36.872 INFO:tasks.kclient:Mounting kernel clients... 2024-04-15T22:57:36.872 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-04-15T22:57:36.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:36.872 DEBUG:teuthology.orchestra.run.smithi037:> ip netns list 2024-04-15T22:57:36.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:36.889 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link delete ceph-brx 2024-04-15T22:57:36.961 INFO:teuthology.orchestra.run.smithi037.stderr:Cannot find device "ceph-brx" 2024-04-15T22:57:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:57:36.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:36.963 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-15T22:57:36.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:36.982 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-04-15T22:57:37.057 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-04-15T22:57:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:57:37.059 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-15T22:57:37.059 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T22:57:37.059 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:37.059 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-15T22:57:37.272 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:37.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-15T22:57:37.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2343435323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-15T22:57:37.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/583780950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-15T22:57:37.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:37.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:37.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:37.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:37.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:37 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2343435323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/583780950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:37.715 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:37 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:38.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.791+0000 7f9e710c8700 1 -- 172.21.15.37:0/3173648756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 msgr2=0x7f9e6c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:38.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3173648756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c0ff790 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009910 tx=0x7f9e5c009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:38.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 -- 172.21.15.37:0/3173648756 shutdown_connections 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3173648756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6c0ffd60 0x7f9e6c103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3173648756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c0ff790 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 -- 172.21.15.37:0/3173648756 >> 172.21.15.37:0/3173648756 conn(0x7f9e6c0fac80 msgr2=0x7f9e6c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 -- 172.21.15.37:0/3173648756 shutdown_connections 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 -- 172.21.15.37:0/3173648756 wait complete. 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.792+0000 7f9e710c8700 1 Processor -- start 2024-04-15T22:57:38.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e710c8700 1 -- start start 2024-04-15T22:57:38.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e710c8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:38.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e710c8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6c0ffd60 0x7f9e6c196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:38.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e710c8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c196b40 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e710c8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c19a8d0 con 0x7f9e6c0ffd60 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58858/0 (socket says 172.21.15.37:58858) 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 -- 172.21.15.37:0/3316639088 learned_addr learned my addr 172.21.15.37:0/3316639088 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6a59c700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6c0ffd60 0x7f9e6c196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 -- 172.21.15.37:0/3316639088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6c0ffd60 msgr2=0x7f9e6c196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6c0ffd60 0x7f9e6c196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 -- 172.21.15.37:0/3316639088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0095f0 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.793+0000 7f9e6ad9d700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c195f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c006010 tx=0x7f9e5c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.794+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e5c01d070 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.794+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c19ab50 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.794+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9e5c004f50 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.794+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e5c021410 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.794+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c19b040 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.795+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c1086e0 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.795+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f9e5c004aa0 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.796+0000 7f9e53fff700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f9e5406c680 0x7f9e5406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.796+0000 7f9e6a59c700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f9e5406c680 0x7f9e5406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.796+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9e5c08cb00 con 0x7f9e6c0ff3c0 2024-04-15T22:57:38.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.796+0000 7f9e6a59c700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f9e5406c680 0x7f9e5406eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c197540 tx=0x7f9e600094f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:38.800+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9e5c057180 con 0x7f9e6c0ff3c0 2024-04-15T22:57:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.049+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9e6c1080c0 con 0x7f9e6c0ff3c0 2024-04-15T22:57:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.049+0000 7f9e53fff700 1 -- 172.21.15.37:0/3316639088 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f9e5c026070 con 0x7f9e6c0ff3c0 2024-04-15T22:57:39.042 INFO:teuthology.orchestra.run.smithi037.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f9e5406c680 msgr2=0x7f9e5406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f9e5406c680 0x7f9e5406eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c197540 tx=0x7f9e600094f0 comp rx=0 tx=0).stop 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 msgr2=0x7f9e6c195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c195f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c006010 tx=0x7f9e5c00be20 comp rx=0 tx=0).stop 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 shutdown_connections 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f9e5406c680 0x7f9e5406eb30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e6c0ff3c0 0x7f9e6c195f20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 --2- 172.21.15.37:0/3316639088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6c0ffd60 0x7f9e6c196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 >> 172.21.15.37:0/3316639088 conn(0x7f9e6c0fac80 msgr2=0x7f9e6c069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 shutdown_connections 2024-04-15T22:57:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39.052+0000 7f9e710c8700 1 -- 172.21.15.37:0/3316639088 wait complete. 2024-04-15T22:57:39.671 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:39 smithi037 ceph-mon[29191]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-15T22:57:39.672 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:39 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3316639088' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-15T22:57:39.672 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi037.front.sepia.ceph.com 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T22:57:39.673 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T22:57:39.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:39.673 DEBUG:teuthology.orchestra.run.smithi037:> ip addr 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: inet6 ::1/128 scope host 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:6b:fd:18 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp7s0f0 2024-04-15T22:57:39.691 INFO:teuthology.orchestra.run.smithi037.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:6b:fd:19 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp7s0f1 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:88:77:94 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp1s0f0 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: inet 172.21.15.37/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft 42235sec preferred_lft 42235sec 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: inet6 fe80::ec4:7aff:fe88:7794/64 scope link noprefixroute 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: valid_lft forever preferred_lft forever 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: link/ether 0c:c4:7a:88:77:95 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:39.692 INFO:teuthology.orchestra.run.smithi037.stdout: altname enp1s0f1 2024-04-15T22:57:39.693 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link add name ceph-brx type bridge 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip addr flush dev ceph-brx 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link set ceph-brx up 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T22:57:39.693 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-04-15T22:57:39.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:39 smithi189 ceph-mon[36179]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-15T22:57:39.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:39 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3316639088' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-15T22:57:39.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:39.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:39.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:39.891 DEBUG:teuthology.orchestra.run.smithi037:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T22:57:39.919 INFO:teuthology.orchestra.run.smithi037.stdout:1 2024-04-15T22:57:39.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:39.920 DEBUG:teuthology.orchestra.run.smithi037:> route 2024-04-15T22:57:39.989 INFO:teuthology.orchestra.run.smithi037.stdout:Kernel IP routing table 2024-04-15T22:57:39.990 INFO:teuthology.orchestra.run.smithi037.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T22:57:39.990 INFO:teuthology.orchestra.run.smithi037.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T22:57:39.990 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T22:57:39.990 INFO:teuthology.orchestra.run.smithi037.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T22:57:39.990 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:39.991 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-04-15T22:57:39.991 DEBUG:teuthology.orchestra.run.smithi037:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T22:57:39.991 DEBUG:teuthology.orchestra.run.smithi037:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T22:57:39.991 DEBUG:teuthology.orchestra.run.smithi037:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T22:57:39.991 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-04-15T22:57:40.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:40.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:40.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:40.323 DEBUG:teuthology.orchestra.run.smithi037:> ip netns list 2024-04-15T22:57:40.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:40.339 DEBUG:teuthology.orchestra.run.smithi037:> ip netns list-id 2024-04-15T22:57:40.396 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:40.396 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-04-15T22:57:40.396 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T22:57:40.396 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-15T22:57:40.396 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-04-15T22:57:40.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:40.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:40.522 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-15T22:57:40.522 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:40.522 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-04-15T22:57:40.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-15T22:57:40.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T22:57:40.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-15T22:57:40.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-15T22:57:40.523 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-15T22:57:40.523 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-04-15T22:57:40.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:40.663 DEBUG:teuthology.orchestra.run.smithi037:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:40.663 DEBUG:teuthology.orchestra.run.smithi037:> set -e 2024-04-15T22:57:40.663 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link set brx.0 up 2024-04-15T22:57:40.663 DEBUG:teuthology.orchestra.run.smithi037:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T22:57:40.663 DEBUG:teuthology.orchestra.run.smithi037:> ') 2024-04-15T22:57:40.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:40.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:40.748 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T22:57:40.748 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:40.763 INFO:teuthology.orchestra.run.smithi037.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T22:57:40.763 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T22:57:40.763 DEBUG:teuthology.orchestra.run.smithi037:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:40.818 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-15T22:57:40.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:40.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-15T22:57:40.921 INFO:teuthology.orchestra.run.smithi037.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-15T22:57:40.922 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-15T22:57:40.922 INFO:teuthology.orchestra.run.smithi037.stdout:invalid new device string format 2024-04-15T22:57:41.042 INFO:teuthology.orchestra.run.smithi037.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-15T22:57:41.042 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-15T22:57:41.042 INFO:teuthology.orchestra.run.smithi037.stdout:invalid new device string format 2024-04-15T22:57:41.042 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: resolved to: "172.21.15.37:3300,172.21.15.189:3300" 2024-04-15T22:57:41.042 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-15T22:57:41.053 INFO:teuthology.orchestra.run.smithi037.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:invalid new device string format 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: resolved to: "172.21.15.37:3300,172.21.15.189:3300" 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: resolved to: "172.21.15.37:6789,172.21.15.189:6789" 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.37:6789,172.21.15.189:6789:/ 2024-04-15T22:57:41.054 INFO:teuthology.orchestra.run.smithi037.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=60ff499a-fb7a-11ee-bc8f-c7b262605968" will pass to kernel 2024-04-15T22:57:41.055 INFO:teuthology.orchestra.run.smithi037.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-15T22:57:41.055 INFO:teuthology.orchestra.run.smithi037.stdout: You just mounted a file system that supports labels which does not 2024-04-15T22:57:41.055 INFO:teuthology.orchestra.run.smithi037.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-15T22:57:41.055 INFO:teuthology.orchestra.run.smithi037.stdout: applications will generate AVC messages and not be allowed access to 2024-04-15T22:57:41.055 INFO:teuthology.orchestra.run.smithi037.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-15T22:57:41.057 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-15T22:57:41.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:41.057 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:41.089 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-15T22:57:41.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T22:57:41.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:41.191 INFO:teuthology.orchestra.run.smithi037.stdout:client24337 2024-04-15T22:57:41.191 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:57:41.192 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T22:57:41.209 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-15T22:57:41.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T22:57:41.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:41.311 INFO:teuthology.orchestra.run.smithi037.stdout:client24337 2024-04-15T22:57:41.311 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-15T22:57:41.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T22:57:41.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:41.372 INFO:teuthology.orchestra.run.smithi037.stdout:client24337 2024-04-15T22:57:41.372 DEBUG:teuthology.orchestra.run.smithi037:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968.client24337/status') 2024-04-15T22:57:41.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:41.440 INFO:teuthology.orchestra.run.smithi037.stdout:instance: client.24337 (3)172.21.15.37:0/60046327 2024-04-15T22:57:41.440 INFO:teuthology.orchestra.run.smithi037.stdout:blocklisted: false 2024-04-15T22:57:41.440 INFO:teuthology.orchestra.run.smithi037.stderr:0+1 records in 2024-04-15T22:57:41.440 INFO:teuthology.orchestra.run.smithi037.stderr:0+1 records out 2024-04-15T22:57:41.440 INFO:teuthology.orchestra.run.smithi037.stderr:69 bytes copied, 2.0952e-05 s, 3.3 MB/s 2024-04-15T22:57:41.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:41.446 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-15T22:57:41.446 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T22:57:41.446 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:41.446 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-15T22:57:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:41.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:41 smithi189 ceph-mon[36179]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-15T22:57:41.782 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:41 smithi037 ceph-mon[29191]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-15T22:57:43.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 -- 172.21.15.37:0/3545743862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2600ffbd0 msgr2=0x7fb260100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:43.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 --2- 172.21.15.37:0/3545743862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2600ffbd0 0x7fb260100040 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb254009910 tx=0x7fb254009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:43.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 -- 172.21.15.37:0/3545743862 shutdown_connections 2024-04-15T22:57:43.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 --2- 172.21.15.37:0/3545743862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2600ffbd0 0x7fb260100040 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:43.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 --2- 172.21.15.37:0/3545743862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb260104130 0x7fb260104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:43.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 -- 172.21.15.37:0/3545743862 >> 172.21.15.37:0/3545743862 conn(0x7fb260075130 msgr2=0x7fb260075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:43.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 -- 172.21.15.37:0/3545743862 shutdown_connections 2024-04-15T22:57:43.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.301+0000 7fb267bad700 1 -- 172.21.15.37:0/3545743862 wait complete. 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb267bad700 1 Processor -- start 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb267bad700 1 -- start start 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb267bad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2600ffbd0 0x7fb260194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb267bad700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 0x7fb260195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb267bad700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb260195950 con 0x7fb260104130 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb267bad700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2601996e0 con 0x7fb2600ffbd0 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.302+0000 7fb265148700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 0x7fb260195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265148700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 0x7fb260195270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49162/0 (socket says 172.21.15.37:49162) 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265148700 1 -- 172.21.15.37:0/4248440300 learned_addr learned my addr 172.21.15.37:0/4248440300 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265949700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2600ffbd0 0x7fb260194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265148700 1 -- 172.21.15.37:0/4248440300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2600ffbd0 msgr2=0x7fb260194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:43.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265148700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2600ffbd0 0x7fb260194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265148700 1 -- 172.21.15.37:0/4248440300 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2540095f0 con 0x7fb260104130 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb265148700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 0x7fb260195270 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb25400f690 tx=0x7fb25400f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb25401d070 con 0x7fb260104130 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb260199960 con 0x7fb260104130 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb254005dc0 con 0x7fb260104130 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2540226a0 con 0x7fb260104130 2024-04-15T22:57:43.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.303+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb260199e50 con 0x7fb260104130 2024-04-15T22:57:43.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.304+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26004f2a0 con 0x7fb260104130 2024-04-15T22:57:43.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.305+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb254003680 con 0x7fb260104130 2024-04-15T22:57:43.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.305+0000 7fb25effd700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb24806c680 0x7fb24806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:43.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.305+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb25408cf40 con 0x7fb260104130 2024-04-15T22:57:43.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.305+0000 7fb265949700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb24806c680 0x7fb24806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:43.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.306+0000 7fb265949700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb24806c680 0x7fb24806eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb250009380 tx=0x7fb25000b040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:43.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.310+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb2540bf720 con 0x7fb260104130 2024-04-15T22:57:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.556+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb26019a070 con 0x7fb260104130 2024-04-15T22:57:43.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.556+0000 7fb25effd700 1 -- 172.21.15.37:0/4248440300 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7fb2540575c0 con 0x7fb260104130 2024-04-15T22:57:43.548 INFO:teuthology.orchestra.run.smithi037.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-15T22:57:43.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb24806c680 msgr2=0x7fb24806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:43.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb24806c680 0x7fb24806eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb250009380 tx=0x7fb25000b040 comp rx=0 tx=0).stop 2024-04-15T22:57:43.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 msgr2=0x7fb260195270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:43.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 0x7fb260195270 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb25400f690 tx=0x7fb25400f770 comp rx=0 tx=0).stop 2024-04-15T22:57:43.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 shutdown_connections 2024-04-15T22:57:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb24806c680 0x7fb24806eb30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2600ffbd0 0x7fb260194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 --2- 172.21.15.37:0/4248440300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb260104130 0x7fb260195270 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 >> 172.21.15.37:0/4248440300 conn(0x7fb260075130 msgr2=0x7fb2600feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 shutdown_connections 2024-04-15T22:57:43.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:43.558+0000 7fb267bad700 1 -- 172.21.15.37:0/4248440300 wait complete. 2024-04-15T22:57:43.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:43 smithi189 ceph-mon[36179]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-04-15T22:57:43.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:43 smithi037 ceph-mon[29191]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-04-15T22:57:44.138 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T22:57:44.139 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T22:57:44.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:44.140 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-15T22:57:44.160 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T22:57:44.160 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T22:57:44.161 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42249sec preferred_lft 42249sec 2024-04-15T22:57:44.162 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::ec4:7aff:fe88:6df1/64 scope link 2024-04-15T22:57:44.163 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-15T22:57:44.163 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T22:57:44.163 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:44.163 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-15T22:57:44.163 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add name ceph-brx type bridge 2024-04-15T22:57:44.163 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr flush dev ceph-brx 2024-04-15T22:57:44.164 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set ceph-brx up 2024-04-15T22:57:44.164 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T22:57:44.164 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-15T22:57:44.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:44.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:44.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:44.373 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T22:57:44.401 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-15T22:57:44.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:44.402 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-15T22:57:44.467 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-15T22:57:44.467 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T22:57:44.468 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T22:57:44.468 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T22:57:44.468 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T22:57:44.468 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:44.468 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-15T22:57:44.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T22:57:44.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T22:57:44.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T22:57:44.468 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-15T22:57:44.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:44.568 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:44 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/4248440300' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-15T22:57:44.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:44.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:44.781 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-15T22:57:44.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:44.799 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-04-15T22:57:44.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:44 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/4248440300' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-15T22:57:44.856 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:44.856 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-15T22:57:44.856 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T22:57:44.856 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-15T22:57:44.856 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-15T22:57:44.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:44.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:44.989 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-15T22:57:44.990 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-15T22:57:45.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:45.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:45.135 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T22:57:45.135 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-15T22:57:45.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-15T22:57:45.136 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T22:57:45.136 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-15T22:57:45.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:45.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:45.221 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T22:57:45.222 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:45.238 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T22:57:45.238 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T22:57:45.238 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:45.293 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-15T22:57:45.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:45.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-15T22:57:45.403 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-15T22:57:45.403 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-15T22:57:45.403 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-15T22:57:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-15T22:57:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-15T22:57:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-15T22:57:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.37:3300,172.21.15.189:3300" 2024-04-15T22:57:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.37:3300,172.21.15.189:3300" 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.37:6789,172.21.15.189:6789" 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.37:6789,172.21.15.189:6789:/ 2024-04-15T22:57:45.521 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=60ff499a-fb7a-11ee-bc8f-c7b262605968" will pass to kernel 2024-04-15T22:57:45.522 INFO:teuthology.orchestra.run.smithi189.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-15T22:57:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: You just mounted a file system that supports labels which does not 2024-04-15T22:57:45.522 INFO:teuthology.orchestra.run.smithi189.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-15T22:57:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: applications will generate AVC messages and not be allowed access to 2024-04-15T22:57:45.523 INFO:teuthology.orchestra.run.smithi189.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-15T22:57:45.524 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-15T22:57:45.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:57:45.525 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:45.553 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-15T22:57:45.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T22:57:45.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:45.651 INFO:teuthology.orchestra.run.smithi189.stdout:client24343 2024-04-15T22:57:45.651 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:57:45.651 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T22:57:45.669 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-15T22:57:45.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:45 smithi189 ceph-mon[36179]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-15T22:57:45.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T22:57:45.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:45.766 INFO:teuthology.orchestra.run.smithi189.stdout:client24343 2024-04-15T22:57:45.766 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-15T22:57:45.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T22:57:45.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:45 smithi037 ceph-mon[29191]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-15T22:57:45.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:45.822 INFO:teuthology.orchestra.run.smithi189.stdout:client24343 2024-04-15T22:57:45.822 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968.client24343/status') 2024-04-15T22:57:45.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T22:57:45.886 INFO:teuthology.orchestra.run.smithi189.stdout:instance: client.24343 (3)192.168.0.1:0/1557948092 2024-04-15T22:57:45.886 INFO:teuthology.orchestra.run.smithi189.stdout:blocklisted: false 2024-04-15T22:57:45.886 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2024-04-15T22:57:45.886 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2024-04-15T22:57:45.886 INFO:teuthology.orchestra.run.smithi189.stderr:70 bytes copied, 2.1069e-05 s, 3.3 MB/s 2024-04-15T22:57:45.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-15T22:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T22:57:45.893 INFO:teuthology.run_tasks:Running task print... 2024-04-15T22:57:45.905 INFO:teuthology.task.print:**** done client 2024-04-15T22:57:45.905 INFO:teuthology.run_tasks:Running task parallel... 2024-04-15T22:57:46.311 INFO:teuthology.task.parallel:starting parallel... 2024-04-15T22:57:46.312 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-15T22:57:46.312 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-15T22:57:46.313 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:46.313 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-04-15T22:57:46.313 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-15T22:57:46.314 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-15T22:57:46.318 INFO:tasks.workunit:Pulling workunits from ref 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 2024-04-15T22:57:46.319 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-15T22:57:46.319 INFO:tasks.workunit:timeout=3h 2024-04-15T22:57:46.319 INFO:tasks.workunit:cleanup=True 2024-04-15T22:57:46.319 DEBUG:teuthology.orchestra.run.smithi037:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 7fh/127d Inode: 1 Links: 2 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-15T22:57:46.341 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-15 22:57:12.367036530 +0000 2024-04-15T22:57:46.342 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-15 22:57:41.092581001 +0000 2024-04-15T22:57:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-04-15 22:57:12.367036530 +0000 2024-04-15T22:57:46.342 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-15T22:57:46.342 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-15T22:57:46.416 DEBUG:teuthology.orchestra.run.smithi189:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:46.434 INFO:teuthology.orchestra.run.smithi189.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:46.434 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-04-15T22:57:46.434 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-04-15T22:57:46.434 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:57:46.434 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-15T22:57:46.434 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-15T22:57:46.435 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-15 22:57:46.419514223 +0000 2024-04-15T22:57:46.435 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-15 22:57:46.419514223 +0000 2024-04-15T22:57:46.435 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-15 22:57:12.367036530 +0000 2024-04-15T22:57:46.435 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-15T22:57:46.435 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-15T22:57:46.511 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 2024-04-15T22:57:46.511 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 2024-04-15T22:57:46.528 INFO:tasks.workunit.client.0.smithi037.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-15T22:57:46.553 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:46.566 INFO:tasks.workunit.client.1.smithi189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-15T22:57:47.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:47 smithi189 ceph-mon[36179]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-15T22:57:47.724 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:47 smithi037 ceph-mon[29191]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-15T22:57:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 -- 172.21.15.37:0/1937435414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec01003d0 msgr2=0x7f2ec0100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/1937435414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec01003d0 0x7f2ec0100840 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009910 tx=0x7f2eb0009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 -- 172.21.15.37:0/1937435414 shutdown_connections 2024-04-15T22:57:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/1937435414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec01003d0 0x7f2ec0100840 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/1937435414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec00ff130 0x7f2ec00ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 -- 172.21.15.37:0/1937435414 >> 172.21.15.37:0/1937435414 conn(0x7f2ec00fa780 msgr2=0x7f2ec00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:48.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 -- 172.21.15.37:0/1937435414 shutdown_connections 2024-04-15T22:57:48.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.060+0000 7f2ec81cd700 1 -- 172.21.15.37:0/1937435414 wait complete. 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec81cd700 1 Processor -- start 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec81cd700 1 -- start start 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec81cd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 0x7f2ec0195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec81cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec01003d0 0x7f2ec0195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec81cd700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec01965c0 con 0x7f2ec00ff130 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec81cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0196700 con 0x7f2ec01003d0 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5f69700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 0x7f2ec0195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5f69700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 0x7f2ec0195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49180/0 (socket says 172.21.15.37:49180) 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5f69700 1 -- 172.21.15.37:0/2063133980 learned_addr learned my addr 172.21.15.37:0/2063133980 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5768700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec01003d0 0x7f2ec0195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:48.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5f69700 1 -- 172.21.15.37:0/2063133980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec01003d0 msgr2=0x7f2ec0195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5f69700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec01003d0 0x7f2ec0195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.061+0000 7f2ec5f69700 1 -- 172.21.15.37:0/2063133980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00095f0 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.062+0000 7f2ec5f69700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 0x7f2ec0195a60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00d8d0 tx=0x7f2ebc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.062+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ebc0098a0 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.062+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2ebc010460 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.062+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec019b1b0 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.062+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ebc00f5f0 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.062+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec019b700 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.063+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec0107c10 con 0x7f2ec00ff130 2024-04-15T22:57:48.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.063+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2ebc009a00 con 0x7f2ec00ff130 2024-04-15T22:57:48.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.064+0000 7f2eb6ffd700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f2eac06c680 0x7f2eac06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:48.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.064+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2ebc08b7c0 con 0x7f2ec00ff130 2024-04-15T22:57:48.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.064+0000 7f2ec5768700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f2eac06c680 0x7f2eac06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:48.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.064+0000 7f2ec5768700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f2eac06c680 0x7f2eac06eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0006010 tx=0x7f2eb000b540 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:48.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.068+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2ebc0598d0 con 0x7f2ec00ff130 2024-04-15T22:57:48.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.280+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f2ec019bd40 con 0x7f2ec00ff130 2024-04-15T22:57:48.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.285+0000 7f2eb6ffd700 1 -- 172.21.15.37:0/2063133980 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7f2ebc059460 con 0x7f2ec00ff130 2024-04-15T22:57:48.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f2eac06c680 msgr2=0x7f2eac06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:48.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f2eac06c680 0x7f2eac06eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0006010 tx=0x7f2eb000b540 comp rx=0 tx=0).stop 2024-04-15T22:57:48.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 msgr2=0x7f2ec0195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:48.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 0x7f2ec0195a60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00d8d0 tx=0x7f2ebc00dbe0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 shutdown_connections 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f2eac06c680 0x7f2eac06eb30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ec00ff130 0x7f2ec0195a60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 --2- 172.21.15.37:0/2063133980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec01003d0 0x7f2ec0195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 >> 172.21.15.37:0/2063133980 conn(0x7f2ec00fa780 msgr2=0x7f2ec0068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 shutdown_connections 2024-04-15T22:57:48.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:48.287+0000 7f2ec81cd700 1 -- 172.21.15.37:0/2063133980 wait complete. 2024-04-15T22:57:48.902 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-15T22:57:48.902 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:48.902 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-15T22:57:49.164 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:49.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2063133980' entity='client.admin' 2024-04-15T22:57:49.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:49.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:49.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:49.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:49.566 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:49.567 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:49 smithi037 ceph-mon[29191]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-15T22:57:49.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2063133980' entity='client.admin' 2024-04-15T22:57:49.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:49.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:57:49.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:57:49.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:49.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:57:49.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:49 smithi189 ceph-mon[36179]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-15T22:57:50.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 -- 172.21.15.37:0/2606223234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 msgr2=0x7fe3d4104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:50.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 --2- 172.21.15.37:0/2606223234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4104730 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009910 tx=0x7fe3c4009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:50.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 -- 172.21.15.37:0/2606223234 shutdown_connections 2024-04-15T22:57:50.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 --2- 172.21.15.37:0/2606223234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4104730 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:50.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 --2- 172.21.15.37:0/2606223234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d40ff990 0x7fe3d4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:50.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 -- 172.21.15.37:0/2606223234 >> 172.21.15.37:0/2606223234 conn(0x7fe3d40f9a50 msgr2=0x7fe3d40fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 -- 172.21.15.37:0/2606223234 shutdown_connections 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 -- 172.21.15.37:0/2606223234 wait complete. 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 Processor -- start 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.650+0000 7fe3dc261700 1 -- start start 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3dc261700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d40ff990 0x7fe3d4197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3dc261700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:50.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3dc261700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d41988b0 con 0x7fe3d41022b0 2024-04-15T22:57:50.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3dc261700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d41989f0 con 0x7fe3d40ff990 2024-04-15T22:57:50.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:50.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58646/0 (socket says 172.21.15.37:58646) 2024-04-15T22:57:50.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 -- 172.21.15.37:0/4139362431 learned_addr learned my addr 172.21.15.37:0/4139362431 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:50.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d9ffd700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d40ff990 0x7fe3d4197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:50.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 -- 172.21.15.37:0/4139362431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d40ff990 msgr2=0x7fe3d4197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:50.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d40ff990 0x7fe3d4197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:50.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 -- 172.21.15.37:0/4139362431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40095f0 con 0x7fe3d41022b0 2024-04-15T22:57:50.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3d97fc700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4198290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4006010 tx=0x7fe3c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:50.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c401d070 con 0x7fe3d41022b0 2024-04-15T22:57:50.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d419d440 con 0x7fe3d41022b0 2024-04-15T22:57:50.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.651+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe3c400f460 con 0x7fe3d41022b0 2024-04-15T22:57:50.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.652+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c4021640 con 0x7fe3d41022b0 2024-04-15T22:57:50.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.652+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d419d930 con 0x7fe3d41022b0 2024-04-15T22:57:50.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.652+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d4108e10 con 0x7fe3d41022b0 2024-04-15T22:57:50.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.653+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe3c400fab0 con 0x7fe3d41022b0 2024-04-15T22:57:50.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.653+0000 7fe3caffd700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe3c006c680 0x7fe3c006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:50.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.654+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe3c408cd70 con 0x7fe3d41022b0 2024-04-15T22:57:50.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.654+0000 7fe3d9ffd700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe3c006c680 0x7fe3c006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:50.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.654+0000 7fe3d9ffd700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe3c006c680 0x7fe3c006eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009a10 tx=0x7fe3d0009190 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:50.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.658+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe3c405ae20 con 0x7fe3d41022b0 2024-04-15T22:57:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.856+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fe3d419e240 con 0x7fe3d41022b0 2024-04-15T22:57:50.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.856+0000 7fe3caffd700 1 -- 172.21.15.37:0/4139362431 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7fe3c405a9b0 con 0x7fe3d41022b0 2024-04-15T22:57:50.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe3c006c680 msgr2=0x7fe3c006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:50.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe3c006c680 0x7fe3c006eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009a10 tx=0x7fe3d0009190 comp rx=0 tx=0).stop 2024-04-15T22:57:50.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 msgr2=0x7fe3d4198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:50.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4198290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4006010 tx=0x7fe3c400bd70 comp rx=0 tx=0).stop 2024-04-15T22:57:50.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 shutdown_connections 2024-04-15T22:57:50.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fe3c006c680 0x7fe3c006eb30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:50.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d40ff990 0x7fe3d4197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:50.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 --2- 172.21.15.37:0/4139362431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3d41022b0 0x7fe3d4198290 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:50.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 >> 172.21.15.37:0/4139362431 conn(0x7fe3d40f9a50 msgr2=0x7fe3d41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:50.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 shutdown_connections 2024-04-15T22:57:50.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:50.858+0000 7fe3dc261700 1 -- 172.21.15.37:0/4139362431 wait complete. 2024-04-15T22:57:51.492 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-15T22:57:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:51.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:51 smithi189 ceph-mon[36179]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:57:51.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:51 smithi037 ceph-mon[29191]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:57:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 -- 172.21.15.37:0/3812005219 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 msgr2=0x7f5340101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 --2- 172.21.15.37:0/3812005219 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340101890 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5334009910 tx=0x7f5334009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 -- 172.21.15.37:0/3812005219 shutdown_connections 2024-04-15T22:57:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 --2- 172.21.15.37:0/3812005219 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340101890 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 --2- 172.21.15.37:0/3812005219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5340100240 0x7f5340100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 -- 172.21.15.37:0/3812005219 >> 172.21.15.37:0/3812005219 conn(0x7f53400fb7f0 msgr2=0x7f53400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 -- 172.21.15.37:0/3812005219 shutdown_connections 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.340+0000 7f5345b17700 1 -- 172.21.15.37:0/3812005219 wait complete. 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f5345b17700 1 Processor -- start 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f5345b17700 1 -- start start 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f5345b17700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5340100240 0x7f5340074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f5345b17700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f5345b17700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5340073680 con 0x7f5340101440 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f5345b17700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400737c0 con 0x7f5340100240 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58664/0 (socket says 172.21.15.37:58664) 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533effd700 1 -- 172.21.15.37:0/3102742863 learned_addr learned my addr 172.21.15.37:0/3102742863 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533f7fe700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5340100240 0x7f5340074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533effd700 1 -- 172.21.15.37:0/3102742863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5340100240 msgr2=0x7f5340074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533effd700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5340100240 0x7f5340074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.341+0000 7f533effd700 1 -- 172.21.15.37:0/3102742863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53340095f0 con 0x7f5340101440 2024-04-15T22:57:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.342+0000 7f533effd700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340073140 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f53340092e0 tx=0x7f533400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.342+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f533401d070 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.342+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5340073a40 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.342+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f533400f460 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.342+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5334021640 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.342+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5340073f30 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.343+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5340108e30 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.343+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f533400fa90 con 0x7f5340101440 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.344+0000 7f533cff9700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f532806c680 0x7f532806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.344+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f533408cca0 con 0x7f5340101440 2024-04-15T22:57:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.344+0000 7f533f7fe700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f532806c680 0x7f532806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.344+0000 7f533f7fe700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f532806c680 0x7f532806eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5330006fd0 tx=0x7f5330008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.349+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5334091070 con 0x7f5340101440 2024-04-15T22:57:53.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.546+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f534019e0e0 con 0x7f5340101440 2024-04-15T22:57:53.539 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:53 smithi037 ceph-mon[29191]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:57:53.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.547+0000 7f533cff9700 1 -- 172.21.15.37:0/3102742863 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7f533405ab20 con 0x7f5340101440 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f532806c680 msgr2=0x7f532806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f532806c680 0x7f532806eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5330006fd0 tx=0x7f5330008040 comp rx=0 tx=0).stop 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 msgr2=0x7f5340073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340073140 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f53340092e0 tx=0x7f533400be20 comp rx=0 tx=0).stop 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 shutdown_connections 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f532806c680 0x7f532806eb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5340100240 0x7f5340074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 --2- 172.21.15.37:0/3102742863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5340101440 0x7f5340073140 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 >> 172.21.15.37:0/3102742863 conn(0x7f53400fb7f0 msgr2=0x7f5340104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 shutdown_connections 2024-04-15T22:57:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:53.549+0000 7f5345b17700 1 -- 172.21.15.37:0/3102742863 wait complete. 2024-04-15T22:57:53.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:53 smithi189 ceph-mon[36179]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:57:54.208 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-15T22:57:54.422 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:55.613 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:55 smithi189 ceph-mon[36179]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:57:55.735 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:55 smithi037 ceph-mon[29191]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:57:56.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 -- 172.21.15.37:0/2701673863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 msgr2=0x7f0d94105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:56.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/2701673863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d94105a80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009870 tx=0x7f0d7c009b80 comp rx=0 tx=0).stop 2024-04-15T22:57:56.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 -- 172.21.15.37:0/2701673863 shutdown_connections 2024-04-15T22:57:56.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/2701673863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d94105a80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:56.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/2701673863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 0x7f0d94103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:56.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 -- 172.21.15.37:0/2701673863 >> 172.21.15.37:0/2701673863 conn(0x7f0d940fa7a0 msgr2=0x7f0d940fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:56.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 -- 172.21.15.37:0/2701673863 shutdown_connections 2024-04-15T22:57:56.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.077+0000 7f0d98dd9700 1 -- 172.21.15.37:0/2701673863 wait complete. 2024-04-15T22:57:56.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.078+0000 7f0d98dd9700 1 Processor -- start 2024-04-15T22:57:56.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.078+0000 7f0d98dd9700 1 -- start start 2024-04-15T22:57:56.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.078+0000 7f0d98dd9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 0x7f0d94191d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:56.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.078+0000 7f0d98dd9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d941922c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:56.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.078+0000 7f0d98dd9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9418e8c0 con 0x7f0d941036a0 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.078+0000 7f0d98dd9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9418ea00 con 0x7f0d94100d80 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d941922c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d941922c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58674/0 (socket says 172.21.15.37:58674) 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 -- 172.21.15.37:0/395959748 learned_addr learned my addr 172.21.15.37:0/395959748 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d9259c700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 0x7f0d94191d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 -- 172.21.15.37:0/395959748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 msgr2=0x7f0d94191d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 0x7f0d94191d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 -- 172.21.15.37:0/395959748 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c009520 con 0x7f0d941036a0 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d9259c700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 0x7f0d94191d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T22:57:56.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d91d9b700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d941922c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009230 tx=0x7f0d7c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:56.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c01c070 con 0x7f0d941036a0 2024-04-15T22:57:56.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0d7c00fe40 con 0x7f0d941036a0 2024-04-15T22:57:56.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9418ec80 con 0x7f0d941036a0 2024-04-15T22:57:56.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.079+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c0173f0 con 0x7f0d941036a0 2024-04-15T22:57:56.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.080+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9418f170 con 0x7f0d941036a0 2024-04-15T22:57:56.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.080+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9404f2a0 con 0x7f0d941036a0 2024-04-15T22:57:56.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.081+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0d7c020070 con 0x7f0d941036a0 2024-04-15T22:57:56.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.082+0000 7f0d8b7fe700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0d8006c680 0x7f0d8006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:56.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.082+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0d7c08cbc0 con 0x7f0d941036a0 2024-04-15T22:57:56.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.082+0000 7f0d9259c700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0d8006c680 0x7f0d8006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:56.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.082+0000 7f0d9259c700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0d8006c680 0x7f0d8006eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d94069890 tx=0x7f0d84009310 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.086+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0d7c05acb0 con 0x7f0d941036a0 2024-04-15T22:57:56.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.289+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f0d941a2480 con 0x7f0d941036a0 2024-04-15T22:57:56.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.289+0000 7f0d8b7fe700 1 -- 172.21.15.37:0/395959748 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f0d7c0270c0 con 0x7f0d941036a0 2024-04-15T22:57:56.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0d8006c680 msgr2=0x7f0d8006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:56.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0d8006c680 0x7f0d8006eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d94069890 tx=0x7f0d84009310 comp rx=0 tx=0).stop 2024-04-15T22:57:56.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 msgr2=0x7f0d941922c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d941922c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009230 tx=0x7f0d7c00f740 comp rx=0 tx=0).stop 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 shutdown_connections 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f0d8006c680 0x7f0d8006eb30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d94100d80 0x7f0d94191d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 --2- 172.21.15.37:0/395959748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d941036a0 0x7f0d941922c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.291+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 >> 172.21.15.37:0/395959748 conn(0x7f0d940fa7a0 msgr2=0x7f0d940ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.292+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 shutdown_connections 2024-04-15T22:57:56.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:56.292+0000 7f0d98dd9700 1 -- 172.21.15.37:0/395959748 wait complete. 2024-04-15T22:57:57.021 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-15T22:57:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:57:57.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:57 smithi037 ceph-mon[29191]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-04-15T22:57:57.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:57 smithi189 ceph-mon[36179]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.064+0000 7f999f07e700 1 -- 172.21.15.37:0/1095717239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99980ff150 msgr2=0x7f99980ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.064+0000 7f999f07e700 1 --2- 172.21.15.37:0/1095717239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99980ff150 0x7f99980ff560 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f998c009910 tx=0x7f998c009c20 comp rx=0 tx=0).stop 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.064+0000 7f999f07e700 1 -- 172.21.15.37:0/1095717239 shutdown_connections 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.064+0000 7f999f07e700 1 --2- 172.21.15.37:0/1095717239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99981003f0 0x7f9998100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.064+0000 7f999f07e700 1 --2- 172.21.15.37:0/1095717239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99980ff150 0x7f99980ff560 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.064+0000 7f999f07e700 1 -- 172.21.15.37:0/1095717239 >> 172.21.15.37:0/1095717239 conn(0x7f99980fa7a0 msgr2=0x7f99980fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 -- 172.21.15.37:0/1095717239 shutdown_connections 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 -- 172.21.15.37:0/1095717239 wait complete. 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 Processor -- start 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 -- start start 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99980ff150 0x7f9998195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 0x7f99981963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99981969c0 con 0x7f99981003f0 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f999f07e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9998196b00 con 0x7f99980ff150 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f9997fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 0x7f99981963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f9997fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 0x7f99981963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58692/0 (socket says 172.21.15.37:58692) 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.065+0000 7f9997fff700 1 -- 172.21.15.37:0/3663786799 learned_addr learned my addr 172.21.15.37:0/3663786799 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f999ce1a700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99980ff150 0x7f9998195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9997fff700 1 -- 172.21.15.37:0/3663786799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99980ff150 msgr2=0x7f9998195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9997fff700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99980ff150 0x7f9998195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9997fff700 1 -- 172.21.15.37:0/3663786799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c0095f0 con 0x7f99981003f0 2024-04-15T22:57:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9997fff700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 0x7f99981963a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f998400b4b0 tx=0x7f998400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9984011800 con 0x7f99981003f0 2024-04-15T22:57:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9984011e10 con 0x7f99981003f0 2024-04-15T22:57:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999819b5b0 con 0x7f99981003f0 2024-04-15T22:57:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f998400f2e0 con 0x7f99981003f0 2024-04-15T22:57:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.066+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999819bb00 con 0x7f99981003f0 2024-04-15T22:57:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.067+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9998068ab0 con 0x7f99981003f0 2024-04-15T22:57:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.068+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f9984010380 con 0x7f99981003f0 2024-04-15T22:57:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.068+0000 7f9995ffb700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f998806c680 0x7f998806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:57:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.068+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f998408b430 con 0x7f99981003f0 2024-04-15T22:57:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.068+0000 7f999ce1a700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f998806c680 0x7f998806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:57:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.069+0000 7f999ce1a700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f998806c680 0x7f998806eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f998c000c00 tx=0x7f998c005fb0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:57:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.073+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9984059670 con 0x7f99981003f0 2024-04-15T22:57:59.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.289+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}) v1 -- 0x7f99980694e0 con 0x7f998806c680 2024-04-15T22:57:59.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.295+0000 7f9995ffb700 1 -- 172.21.15.37:0/3663786799 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f99980694e0 con 0x7f998806c680 2024-04-15T22:57:59.287 INFO:teuthology.orchestra.run.smithi037.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f998806c680 msgr2=0x7f998806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f998806c680 0x7f998806eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f998c000c00 tx=0x7f998c005fb0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 msgr2=0x7f99981963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 0x7f99981963a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f998400b4b0 tx=0x7f998400b870 comp rx=0 tx=0).stop 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 shutdown_connections 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f998806c680 0x7f998806eb30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99980ff150 0x7f9998195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 --2- 172.21.15.37:0/3663786799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99981003f0 0x7f99981963a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 >> 172.21.15.37:0/3663786799 conn(0x7f99980fa7a0 msgr2=0x7f9998068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:57:59.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 shutdown_connections 2024-04-15T22:57:59.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:57:59.297+0000 7f999f07e700 1 -- 172.21.15.37:0/3663786799 wait complete. 2024-04-15T22:57:59.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:59 smithi037 ceph-mon[29191]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-15T22:57:59.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:59 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:59.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:57:59 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:59.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:59 smithi189 ceph-mon[36179]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-15T22:57:59.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:59 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:57:59.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:57:59 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:57:59.957 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-15T22:57:59.957 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi037.front.sepia.ceph.com 2024-04-15T22:57:59.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-15T22:58:00.192 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:00.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:00 smithi037 ceph-mon[29191]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:00.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:00 smithi037 ceph-mon[29191]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:58:00.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:58:00.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:58:00.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:00 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:00.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:00 smithi189 ceph-mon[36179]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:00.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:00 smithi189 ceph-mon[36179]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:58:00.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:58:00.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:58:00.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:00 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:01.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:01 smithi037 ceph-mon[29191]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:58:01.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:01 smithi037 ceph-mon[29191]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-15T22:58:01.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:01 smithi189 ceph-mon[36179]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:58:01.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:01 smithi189 ceph-mon[36179]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-15T22:58:02.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 -- 172.21.15.37:0/3051774543 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4069b00 msgr2=0x7face4102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 --2- 172.21.15.37:0/3051774543 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4069b00 0x7face4102df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7facd4009910 tx=0x7facd4009c20 comp rx=0 tx=0).stop 2024-04-15T22:58:02.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 -- 172.21.15.37:0/3051774543 shutdown_connections 2024-04-15T22:58:02.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 --2- 172.21.15.37:0/3051774543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face4103330 0x7face41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 --2- 172.21.15.37:0/3051774543 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4069b00 0x7face4102df0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 -- 172.21.15.37:0/3051774543 >> 172.21.15.37:0/3051774543 conn(0x7face40fa7a0 msgr2=0x7face40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 -- 172.21.15.37:0/3051774543 shutdown_connections 2024-04-15T22:58:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.119+0000 7facea17c700 1 -- 172.21.15.37:0/3051774543 wait complete. 2024-04-15T22:58:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.120+0000 7facea17c700 1 Processor -- start 2024-04-15T22:58:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7facea17c700 1 -- start start 2024-04-15T22:58:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7facea17c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 0x7face4100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7facea17c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face40fed90 0x7face40ff200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7facea17c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face40ff740 con 0x7face4103330 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7facea17c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face40ff8b0 con 0x7face40fed90 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 0x7face4100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 0x7face4100740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43084/0 (socket says 172.21.15.37:43084) 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 -- 172.21.15.37:0/4038543833 learned_addr learned my addr 172.21.15.37:0/4038543833 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face2ffd700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face40fed90 0x7face40ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 -- 172.21.15.37:0/4038543833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face40fed90 msgr2=0x7face40ff200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face40fed90 0x7face40ff200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 -- 172.21.15.37:0/4038543833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40095f0 con 0x7face4103330 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.121+0000 7face37fe700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 0x7face4100740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7facd400bda0 tx=0x7facd400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.122+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd401d070 con 0x7face4103330 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.122+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face40ffb30 con 0x7face4103330 2024-04-15T22:58:02.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.122+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7facd4004c50 con 0x7face4103330 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.122+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face419e0f0 con 0x7face4103330 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.122+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd400f690 con 0x7face4103330 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.123+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face418fc70 con 0x7face4103330 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.123+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7facd4022a20 con 0x7face4103330 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.124+0000 7face0ff9700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7faccc06c680 0x7faccc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.124+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7facd408cef0 con 0x7face4103330 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.124+0000 7face2ffd700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7faccc06c680 0x7faccc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.124+0000 7face2ffd700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7faccc06c680 0x7faccc06eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facd8006fd0 tx=0x7facd8008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:02.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.132+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7facd405b130 con 0x7face4103330 2024-04-15T22:58:02.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.374+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7face4061190 con 0x7faccc06c680 2024-04-15T22:58:02.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.375+0000 7face0ff9700 1 -- 172.21.15.37:0/4038543833 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7face4061190 con 0x7faccc06c680 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7faccc06c680 msgr2=0x7faccc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7faccc06c680 0x7faccc06eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facd8006fd0 tx=0x7facd8008040 comp rx=0 tx=0).stop 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 msgr2=0x7face4100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 0x7face4100740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7facd400bda0 tx=0x7facd400be80 comp rx=0 tx=0).stop 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 shutdown_connections 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7faccc06c680 0x7faccc06eb30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7face4103330 0x7face4100740 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 --2- 172.21.15.37:0/4038543833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face40fed90 0x7face40ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 >> 172.21.15.37:0/4038543833 conn(0x7face40fa7a0 msgr2=0x7face40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 shutdown_connections 2024-04-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.378+0000 7facea17c700 1 -- 172.21.15.37:0/4038543833 wait complete. 2024-04-15T22:58:02.386 INFO:teuthology.orchestra.run.smithi037.stdout:true 2024-04-15T22:58:02.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/134750480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 msgr2=0x7fdb541057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/134750480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb541057b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009910 tx=0x7fdb48009c20 comp rx=0 tx=0).stop 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/134750480 shutdown_connections 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/134750480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb541057b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/134750480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb54069ac0 0x7fdb54102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/134750480 >> 172.21.15.37:0/134750480 conn(0x7fdb540fa7a0 msgr2=0x7fdb540fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/134750480 shutdown_connections 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.496+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/134750480 wait complete. 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb5a2bc700 1 Processor -- start 2024-04-15T22:58:02.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb5a2bc700 1 -- start start 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb5a2bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb54069ac0 0x7fdb54100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb5a2bc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb540fedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb5a2bc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540ff310 con 0x7fdb54103370 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb5a2bc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540ff450 con 0x7fdb54069ac0 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb537fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb540fedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.497+0000 7fdb537fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb540fedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43108/0 (socket says 172.21.15.37:43108) 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb537fe700 1 -- 172.21.15.37:0/656544978 learned_addr learned my addr 172.21.15.37:0/656544978 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb53fff700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb54069ac0 0x7fdb54100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:02.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb537fe700 1 -- 172.21.15.37:0/656544978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb54069ac0 msgr2=0x7fdb54100780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb537fe700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb54069ac0 0x7fdb54100780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb537fe700 1 -- 172.21.15.37:0/656544978 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb480095f0 con 0x7fdb54103370 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb537fe700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb540fedd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800b5c0 tx=0x7fdb4800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb4801d070 con 0x7fdb54103370 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb540ff6d0 con 0x7fdb54103370 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdb4800f460 con 0x7fdb54103370 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb48021640 con 0x7fdb54103370 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.498+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb540ffbc0 con 0x7fdb54103370 2024-04-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.499+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5418fc10 con 0x7fdb54103370 2024-04-15T22:58:02.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.500+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fdb4800fa60 con 0x7fdb54103370 2024-04-15T22:58:02.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.500+0000 7fdb517fa700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdb3c06c680 0x7fdb3c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:02.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.500+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdb4808cc00 con 0x7fdb54103370 2024-04-15T22:58:02.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.500+0000 7fdb53fff700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdb3c06c680 0x7fdb3c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:02.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.501+0000 7fdb53fff700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdb3c06c680 0x7fdb3c06eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb540697f0 tx=0x7fdb4400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:02.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.505+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdb4805ac70 con 0x7fdb54103370 2024-04-15T22:58:02.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.730+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb5402cf80 con 0x7fdb3c06c680 2024-04-15T22:58:02.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.731+0000 7fdb517fa700 1 -- 172.21.15.37:0/656544978 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fdb5402cf80 con 0x7fdb3c06c680 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdb3c06c680 msgr2=0x7fdb3c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdb3c06c680 0x7fdb3c06eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb540697f0 tx=0x7fdb4400b410 comp rx=0 tx=0).stop 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 msgr2=0x7fdb540fedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb540fedd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800b5c0 tx=0x7fdb4800bd70 comp rx=0 tx=0).stop 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 shutdown_connections 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fdb3c06c680 0x7fdb3c06eb30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb54069ac0 0x7fdb54100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 --2- 172.21.15.37:0/656544978 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb54103370 0x7fdb540fedd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:02.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 >> 172.21.15.37:0/656544978 conn(0x7fdb540fa7a0 msgr2=0x7fdb540fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:02.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 shutdown_connections 2024-04-15T22:58:02.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:02.733+0000 7fdb5a2bc700 1 -- 172.21.15.37:0/656544978 wait complete. 2024-04-15T22:58:02.737 INFO:teuthology.orchestra.run.smithi037.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T22:58:03.470 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T22:58:03.686 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:03.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:03.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:03.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:58:03.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:03.800 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:58:03.801 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:04.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:04 smithi189 ceph-mon[36179]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:04.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:04 smithi037 ceph-mon[29191]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:05.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 -- 172.21.15.37:0/3977073584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 msgr2=0x7fecbc100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:05.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3977073584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc100650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fecac009910 tx=0x7fecac009c20 comp rx=0 tx=0).stop 2024-04-15T22:58:05.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 -- 172.21.15.37:0/3977073584 shutdown_connections 2024-04-15T22:58:05.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3977073584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecbc101440 0x7fecbc101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3977073584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc100650 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 -- 172.21.15.37:0/3977073584 >> 172.21.15.37:0/3977073584 conn(0x7fecbc0fb7f0 msgr2=0x7fecbc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 -- 172.21.15.37:0/3977073584 shutdown_connections 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 -- 172.21.15.37:0/3977073584 wait complete. 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 Processor -- start 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.359+0000 7fecc0abc700 1 -- start start 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecc0abc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecc0abc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecbc101440 0x7fecbc196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecc0abc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc196630 con 0x7fecbc100240 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecc0abc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc196770 con 0x7fecbc101440 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43130/0 (socket says 172.21.15.37:43130) 2024-04-15T22:58:05.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 -- 172.21.15.37:0/3885343941 learned_addr learned my addr 172.21.15.37:0/3885343941 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:58:05.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 -- 172.21.15.37:0/3885343941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecbc101440 msgr2=0x7fecbc196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:05.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecbc101440 0x7fecbc196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 -- 172.21.15.37:0/3885343941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0095f0 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecba59c700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc195ad0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fecac00bd70 tx=0x7fecac00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecac01d070 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc19b1c0 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fecac022470 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.360+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecac00f690 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.361+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecbc19b6b0 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.361+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecbc108d90 con 0x7fecbc100240 2024-04-15T22:58:05.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.362+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fecac0225e0 con 0x7fecbc100240 2024-04-15T22:58:05.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.362+0000 7fecab7fe700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7feca406c680 0x7feca406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:05.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.362+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fecac08cf90 con 0x7fecbc100240 2024-04-15T22:58:05.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.362+0000 7fecb9d9b700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7feca406c680 0x7feca406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:05.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.363+0000 7fecb9d9b700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7feca406c680 0x7feca406eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fecb000a990 tx=0x7fecb0005c80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:05.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.367+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fecac05b1c0 con 0x7fecbc100240 2024-04-15T22:58:05.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.576+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fecbc105d90 con 0x7feca406c680 2024-04-15T22:58:05.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.586+0000 7fecab7fe700 1 -- 172.21.15.37:0/3885343941 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fecbc105d90 con 0x7feca406c680 2024-04-15T22:58:05.578 INFO:teuthology.orchestra.run.smithi037.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:58:05.578 INFO:teuthology.orchestra.run.smithi037.stdout:alertmanager.smithi037 smithi037 *:9093,9094 running (4m) 28s ago 6m 25.2M - 0.25.0 c8568f914cd2 e049eff81a04 2024-04-15T22:58:05.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter.smithi037 smithi037 running (6m) 28s ago 6m 7424k - 18.2.0 dc2bc1663786 3616d122dfb5 2024-04-15T22:58:05.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter.smithi189 smithi189 running (5m) 33s ago 5m 7348k - 18.2.0 dc2bc1663786 4b1aba4b2eba 2024-04-15T22:58:05.578 INFO:teuthology.orchestra.run.smithi037.stdout:crash.smithi037 smithi037 running (6m) 28s ago 6m 6665k - 18.2.0 dc2bc1663786 944063fe7d85 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:crash.smithi189 smithi189 running (5m) 33s ago 5m 6660k - 18.2.0 dc2bc1663786 f483fc8d01e0 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:grafana.smithi037 smithi037 *:3000 running (4m) 28s ago 6m 83.6M - 9.4.7 954c08fa6188 a7a1c1b0e042 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi037.ifpdxj smithi037 running (47s) 28s ago 46s 18.0M - 18.2.0 dc2bc1663786 e360d43528ac 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi037.rdehay smithi037 running (54s) 28s ago 53s 11.3M - 18.2.0 dc2bc1663786 f6a0a7e18f69 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi189.kzokod smithi189 running (44s) 33s ago 44s 12.0M - 18.2.0 dc2bc1663786 9a555801f8f6 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi189.pqjbmb smithi189 running (51s) 33s ago 51s 15.6M - 18.2.0 dc2bc1663786 616697835d13 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.smithi037.gtwkck smithi037 *:9283,8765,8443 running (8m) 28s ago 8m 487M - 18.2.0 dc2bc1663786 9fa350a65979 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.smithi189.lanqnp smithi189 *:8443,9283,8765 running (5m) 33s ago 5m 438M - 18.2.0 dc2bc1663786 a9d94b5f083f 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mon.smithi037 smithi037 running (8m) 28s ago 8m 49.1M 2048M 18.2.0 dc2bc1663786 f39d1d96ec93 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:mon.smithi189 smithi189 running (5m) 33s ago 5m 36.1M 2048M 18.2.0 dc2bc1663786 e2331fdccbaa 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter.smithi037 smithi037 *:9100 running (6m) 28s ago 6m 16.0M - 1.5.0 0da6a335fe13 a4d1b4d06e69 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter.smithi189 smithi189 *:9100 running (5m) 33s ago 5m 15.0M - 1.5.0 0da6a335fe13 f9d0ca18f1f6 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 smithi037 running (4m) 28s ago 4m 48.3M 3948M 18.2.0 dc2bc1663786 45282f17f939 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 smithi037 running (4m) 28s ago 4m 50.7M 3948M 18.2.0 dc2bc1663786 7c4e631bf214 2024-04-15T22:58:05.579 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 smithi037 running (3m) 28s ago 3m 48.6M 3948M 18.2.0 dc2bc1663786 2c1c43054c1a 2024-04-15T22:58:05.580 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 smithi189 running (3m) 33s ago 3m 46.0M 4970M 18.2.0 dc2bc1663786 5e54e0afe09f 2024-04-15T22:58:05.580 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 smithi189 running (2m) 33s ago 2m 44.3M 4970M 18.2.0 dc2bc1663786 c6102fb44c70 2024-04-15T22:58:05.580 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 smithi189 running (2m) 33s ago 2m 46.9M 4970M 18.2.0 dc2bc1663786 e4f59c36ac6f 2024-04-15T22:58:05.580 INFO:teuthology.orchestra.run.smithi037.stdout:prometheus.smithi037 smithi037 *:9095 running (4m) 28s ago 5m 62.5M - 2.43.0 a07b618ecd1d a2075b4529f9 2024-04-15T22:58:05.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7feca406c680 msgr2=0x7feca406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7feca406c680 0x7feca406eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fecb000a990 tx=0x7fecb0005c80 comp rx=0 tx=0).stop 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 msgr2=0x7fecbc195ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc195ad0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fecac00bd70 tx=0x7fecac00be50 comp rx=0 tx=0).stop 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 shutdown_connections 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7feca406c680 0x7feca406eb30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecbc100240 0x7fecbc195ad0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.588+0000 7fecc0abc700 1 --2- 172.21.15.37:0/3885343941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecbc101440 0x7fecbc196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.589+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 >> 172.21.15.37:0/3885343941 conn(0x7fecbc0fb7f0 msgr2=0x7fecbc104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.589+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 shutdown_connections 2024-04-15T22:58:05.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:05.589+0000 7fecc0abc700 1 -- 172.21.15.37:0/3885343941 wait complete. 2024-04-15T22:58:05.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:05 smithi189 ceph-mon[36179]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:05.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:05 smithi037 ceph-mon[29191]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:06.306 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade status' 2024-04-15T22:58:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:06.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:06 smithi037 ceph-mon[29191]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:06.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:06 smithi189 ceph-mon[36179]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:07.646 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:07 smithi037 ceph-mon[29191]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:07.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:07 smithi189 ceph-mon[36179]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 -- 172.21.15.37:0/1071753166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 msgr2=0x7fb704100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 --2- 172.21.15.37:0/1071753166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704100650 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009910 tx=0x7fb6f4009c20 comp rx=0 tx=0).stop 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 -- 172.21.15.37:0/1071753166 shutdown_connections 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 --2- 172.21.15.37:0/1071753166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704101440 0x7fb704101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 --2- 172.21.15.37:0/1071753166 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704100650 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 -- 172.21.15.37:0/1071753166 >> 172.21.15.37:0/1071753166 conn(0x7fb7040fb7f0 msgr2=0x7fb7040fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 -- 172.21.15.37:0/1071753166 shutdown_connections 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.145+0000 7fb709d83700 1 -- 172.21.15.37:0/1071753166 wait complete. 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb709d83700 1 Processor -- start 2024-04-15T22:58:08.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb709d83700 1 -- start start 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb709d83700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb709d83700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704101440 0x7fb7041960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb709d83700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041966c0 con 0x7fb704100240 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb709d83700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb704196800 con 0x7fb704101440 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb7037fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb7037fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43144/0 (socket says 172.21.15.37:43144) 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb7037fe700 1 -- 172.21.15.37:0/3860673708 learned_addr learned my addr 172.21.15.37:0/3860673708 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.146+0000 7fb6faffd700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704101440 0x7fb7041960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7037fe700 1 -- 172.21.15.37:0/3860673708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704101440 msgr2=0x7fb7041960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7037fe700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704101440 0x7fb7041960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7037fe700 1 -- 172.21.15.37:0/3860673708 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40095f0 con 0x7fb704100240 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7037fe700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704195b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400bda0 tx=0x7fb6f400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:08.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6f401d070 con 0x7fb704100240 2024-04-15T22:58:08.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70419b250 con 0x7fb704100240 2024-04-15T22:58:08.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb6f4004c50 con 0x7fb704100240 2024-04-15T22:58:08.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6f400f690 con 0x7fb704100240 2024-04-15T22:58:08.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.147+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70419b740 con 0x7fb704100240 2024-04-15T22:58:08.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.148+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb704108e10 con 0x7fb704100240 2024-04-15T22:58:08.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.148+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb6f40229b0 con 0x7fb704100240 2024-04-15T22:58:08.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.149+0000 7fb7017fa700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb6f006c680 0x7fb6f006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:08.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.149+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb6f408cfc0 con 0x7fb704100240 2024-04-15T22:58:08.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.149+0000 7fb6faffd700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb6f006c680 0x7fb6f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:08.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.149+0000 7fb6faffd700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb6f006c680 0x7fb6f006eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:08.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.154+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb6f405b180 con 0x7fb704100240 2024-04-15T22:58:08.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.372+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb704105d90 con 0x7fb6f006c680 2024-04-15T22:58:08.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.374+0000 7fb7017fa700 1 -- 172.21.15.37:0/3860673708 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fb704105d90 con 0x7fb6f006c680 2024-04-15T22:58:08.365 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-15T22:58:08.365 INFO:teuthology.orchestra.run.smithi037.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", 2024-04-15T22:58:08.365 INFO:teuthology.orchestra.run.smithi037.stdout: "in_progress": true, 2024-04-15T22:58:08.365 INFO:teuthology.orchestra.run.smithi037.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-15T22:58:08.365 INFO:teuthology.orchestra.run.smithi037.stdout: "services_complete": [], 2024-04-15T22:58:08.366 INFO:teuthology.orchestra.run.smithi037.stdout: "progress": "", 2024-04-15T22:58:08.366 INFO:teuthology.orchestra.run.smithi037.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", 2024-04-15T22:58:08.366 INFO:teuthology.orchestra.run.smithi037.stdout: "is_paused": true 2024-04-15T22:58:08.366 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb6f006c680 msgr2=0x7fb6f006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb6f006c680 0x7fb6f006eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).stop 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 msgr2=0x7fb704195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704195b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400bda0 tx=0x7fb6f400be80 comp rx=0 tx=0).stop 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 shutdown_connections 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb6f006c680 0x7fb6f006eb30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704100240 0x7fb704195b60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 --2- 172.21.15.37:0/3860673708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704101440 0x7fb7041960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 >> 172.21.15.37:0/3860673708 conn(0x7fb7040fb7f0 msgr2=0x7fb704104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 shutdown_connections 2024-04-15T22:58:08.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:08.376+0000 7fb709d83700 1 -- 172.21.15.37:0/3860673708 wait complete. 2024-04-15T22:58:09.109 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph health detail' 2024-04-15T22:58:09.326 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:09.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:09 smithi189 ceph-mon[36179]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:09.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:09 smithi189 ceph-mon[36179]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:09.756 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:09 smithi037 ceph-mon[29191]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:58:09.756 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:09 smithi037 ceph-mon[29191]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 -- 172.21.15.37:0/1288458689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c100240 msgr2=0x7f4f3c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 --2- 172.21.15.37:0/1288458689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c100240 0x7f4f3c100650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009910 tx=0x7f4f30009c20 comp rx=0 tx=0).stop 2024-04-15T22:58:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 -- 172.21.15.37:0/1288458689 shutdown_connections 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 --2- 172.21.15.37:0/1288458689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c101440 0x7f4f3c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 --2- 172.21.15.37:0/1288458689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c100240 0x7f4f3c100650 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 -- 172.21.15.37:0/1288458689 >> 172.21.15.37:0/1288458689 conn(0x7f4f3c0fb7f0 msgr2=0x7f4f3c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 -- 172.21.15.37:0/1288458689 shutdown_connections 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.978+0000 7f4f43366700 1 -- 172.21.15.37:0/1288458689 wait complete. 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f43366700 1 Processor -- start 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f43366700 1 -- start start 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f43366700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c100240 0x7f4f3c195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f43366700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 0x7f4f3c1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f43366700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1966e0 con 0x7f4f3c101440 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f43366700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c196820 con 0x7f4f3c100240 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 0x7f4f3c1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 0x7f4f3c1960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37128/0 (socket says 172.21.15.37:37128) 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 -- 172.21.15.37:0/554825185 learned_addr learned my addr 172.21.15.37:0/554825185 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f41102700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c100240 0x7f4f3c195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 -- 172.21.15.37:0/554825185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c100240 msgr2=0x7f4f3c195b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c100240 0x7f4f3c195b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 -- 172.21.15.37:0/554825185 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f300095f0 con 0x7f4f3c101440 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f41102700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c100240 0x7f4f3c195b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.979+0000 7f4f40901700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 0x7f4f3c1960c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800b4b0 tx=0x7f4f3800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.980+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f38011800 con 0x7f4f3c101440 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.980+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4f38011e10 con 0x7f4f3c101440 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.980+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c19b2d0 con 0x7f4f3c101440 2024-04-15T22:58:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.980+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f3800f2e0 con 0x7f4f3c101440 2024-04-15T22:58:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.980+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c19b820 con 0x7f4f3c101440 2024-04-15T22:58:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.981+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c108e10 con 0x7f4f3c101440 2024-04-15T22:58:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.981+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f4f38010380 con 0x7f4f3c101440 2024-04-15T22:58:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.982+0000 7f4f2e7fc700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f4f2806c680 0x7f4f2806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.982+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4f3808b430 con 0x7f4f3c101440 2024-04-15T22:58:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.982+0000 7f4f41102700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f4f2806c680 0x7f4f2806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.982+0000 7f4f41102700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f4f2806c680 0x7f4f2806eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4f30006010 tx=0x7f4f30005960 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:10.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:10.986+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4f38059670 con 0x7f4f3c101440 2024-04-15T22:58:11.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.266+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4f3c108fd0 con 0x7f4f3c101440 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.266+0000 7f4f2e7fc700 1 -- 172.21.15.37:0/554825185 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1054 (secure 0 0 0) 0x7f4f38059200 con 0x7f4f3c101440 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; Upgrade: failed to pull target image 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stdout: host smithi037 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T22:58:11.258 INFO:teuthology.orchestra.run.smithi037.stdout:Non-zero exit code 125 from /usr/bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:58:11.259 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T22:58:11.259 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765: reading manifest a9a752df26c63acad72e1b3569fd79a515ca0765 in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-15T22:58:11.259 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: Failed command: /usr/bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:58:11.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f4f2806c680 msgr2=0x7f4f2806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f4f2806c680 0x7f4f2806eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4f30006010 tx=0x7f4f30005960 comp rx=0 tx=0).stop 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 msgr2=0x7f4f3c1960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 0x7f4f3c1960c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800b4b0 tx=0x7f4f3800b870 comp rx=0 tx=0).stop 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 shutdown_connections 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f4f2806c680 0x7f4f2806eb30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f3c100240 0x7f4f3c195b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 --2- 172.21.15.37:0/554825185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f3c101440 0x7f4f3c1960c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 >> 172.21.15.37:0/554825185 conn(0x7f4f3c0fb7f0 msgr2=0x7f4f3c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 shutdown_connections 2024-04-15T22:58:11.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:11.268+0000 7f4f43366700 1 -- 172.21.15.37:0/554825185 wait complete. 2024-04-15T22:58:11.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:11 smithi037 ceph-mon[29191]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:11.565 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:11 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/554825185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:58:11.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:11 smithi189 ceph-mon[36179]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:11.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:11 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/554825185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:58:11.914 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T22:58:12.127 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:13.419 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:13 smithi037 ceph-mon[29191]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:13.714 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:13 smithi189 ceph-mon[36179]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:13.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 -- 172.21.15.37:0/2242127118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4641014e0 msgr2=0x7fb464101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:13.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 --2- 172.21.15.37:0/2242127118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4641014e0 0x7fb464101950 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009910 tx=0x7fb44c009c20 comp rx=0 tx=0).stop 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 -- 172.21.15.37:0/2242127118 shutdown_connections 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 --2- 172.21.15.37:0/2242127118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4641014e0 0x7fb464101950 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 --2- 172.21.15.37:0/2242127118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb464100240 0x7fb464100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 -- 172.21.15.37:0/2242127118 >> 172.21.15.37:0/2242127118 conn(0x7fb4640fb890 msgr2=0x7fb4640fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 -- 172.21.15.37:0/2242127118 shutdown_connections 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 -- 172.21.15.37:0/2242127118 wait complete. 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.823+0000 7fb469ed2700 1 Processor -- start 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb469ed2700 1 -- start start 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb469ed2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 0x7fb464074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb469ed2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4641014e0 0x7fb464073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb469ed2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb464073680 con 0x7fb464100240 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb469ed2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4640737c0 con 0x7fb4641014e0 2024-04-15T22:58:13.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 0x7fb464074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 0x7fb464074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37156/0 (socket says 172.21.15.37:37156) 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 -- 172.21.15.37:0/1761230314 learned_addr learned my addr 172.21.15.37:0/1761230314 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb462ffd700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4641014e0 0x7fb464073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 -- 172.21.15.37:0/1761230314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4641014e0 msgr2=0x7fb464073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4641014e0 0x7fb464073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 -- 172.21.15.37:0/1761230314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c0095f0 con 0x7fb464100240 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb462ffd700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4641014e0 0x7fb464073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.824+0000 7fb4637fe700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 0x7fb464074af0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb45400d8d0 tx=0x7fb45400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.825+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb454009940 con 0x7fb464100240 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.825+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb454010460 con 0x7fb464100240 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.825+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb464073aa0 con 0x7fb464100240 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.825+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb45400f5f0 con 0x7fb464100240 2024-04-15T22:58:13.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.825+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46419daf0 con 0x7fb464100240 2024-04-15T22:58:13.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.826+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb464108d20 con 0x7fb464100240 2024-04-15T22:58:13.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.826+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb4540105d0 con 0x7fb464100240 2024-04-15T22:58:13.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.827+0000 7fb460ff9700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb45006c680 0x7fb45006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:58:13.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.827+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb45408b8b0 con 0x7fb464100240 2024-04-15T22:58:13.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.827+0000 7fb462ffd700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb45006c680 0x7fb45006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:58:13.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.827+0000 7fb462ffd700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb45006c680 0x7fb45006eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00b5c0 tx=0x7fb44c005e20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:58:13.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:13.831+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb4540599a0 con 0x7fb464100240 2024-04-15T22:58:14.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.112+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb46419e0a0 con 0x7fb464100240 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.113+0000 7fb460ff9700 1 -- 172.21.15.37:0/1761230314 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fb45405de60 con 0x7fb464100240 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": { 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": { 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-15T22:58:14.105 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": { 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "mds": { 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "overall": { 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-15T22:58:14.106 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb45006c680 msgr2=0x7fb45006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb45006c680 0x7fb45006eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00b5c0 tx=0x7fb44c005e20 comp rx=0 tx=0).stop 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 msgr2=0x7fb464074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 0x7fb464074af0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb45400d8d0 tx=0x7fb45400dc90 comp rx=0 tx=0).stop 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 shutdown_connections 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fb45006c680 0x7fb45006eb30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:14.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb464100240 0x7fb464074af0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:14.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 --2- 172.21.15.37:0/1761230314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4641014e0 0x7fb464073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:58:14.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.115+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 >> 172.21.15.37:0/1761230314 conn(0x7fb4640fb890 msgr2=0x7fb464104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:58:14.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.116+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 shutdown_connections 2024-04-15T22:58:14.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:58:14.116+0000 7fb469ed2700 1 -- 172.21.15.37:0/1761230314 wait complete. 2024-04-15T22:58:14.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:14 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1761230314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:58:14.782 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:14 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1761230314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:58:14.806 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-15T22:58:15.021 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:15.453 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:15 smithi037 ceph-mon[29191]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:15.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:15 smithi189 ceph-mon[36179]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:16.786 INFO:teuthology.orchestra.run.smithi037.stdout:wait for servicemap items w/ changing names to refresh 2024-04-15T22:58:17.572 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'sleep 60' 2024-04-15T22:58:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:58:17.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:17 smithi037 ceph-mon[29191]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:17.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:17 smithi189 ceph-mon[36179]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:18.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:18 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:18.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:18 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:19.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:19 smithi037 ceph-mon[29191]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:19.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:19 smithi189 ceph-mon[36179]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:21.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:21 smithi037 ceph-mon[29191]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:21.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:21 smithi189 ceph-mon[36179]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:23.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:23 smithi189 ceph-mon[36179]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:23.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:23 smithi037 ceph-mon[29191]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:25.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:25 smithi037 ceph-mon[29191]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:25.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:25 smithi189 ceph-mon[36179]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:27.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:27 smithi037 ceph-mon[29191]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-15T22:58:27.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:27 smithi189 ceph-mon[36179]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-15T22:58:29.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:29 smithi037 ceph-mon[29191]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-15T22:58:29.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:29 smithi189 ceph-mon[36179]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-15T22:58:31.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:31 smithi037 ceph-mon[29191]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-15T22:58:31.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:31 smithi189 ceph-mon[36179]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-15T22:58:33.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:33 smithi189 ceph-mon[36179]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:33.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:33 smithi037 ceph-mon[29191]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:34.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:34 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:34.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:34 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:35.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:35 smithi189 ceph-mon[36179]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:35.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:35 smithi037 ceph-mon[29191]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:37.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:37 smithi037 ceph-mon[29191]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:37.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:37 smithi189 ceph-mon[36179]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-15T22:58:39.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:39 smithi037 ceph-mon[29191]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-15T22:58:39.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:39 smithi189 ceph-mon[36179]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-15T22:58:41.817 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:41 smithi037 ceph-mon[29191]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-15T22:58:41.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:41 smithi189 ceph-mon[36179]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-15T22:58:43.471 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:43 smithi037 ceph-mon[29191]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:43.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:43 smithi189 ceph-mon[36179]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:45.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:45 smithi037 ceph-mon[29191]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:45.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:45 smithi189 ceph-mon[36179]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:47.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:47 smithi037 ceph-mon[29191]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:47.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:47 smithi189 ceph-mon[36179]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:48.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:48 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:48.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:48 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:58:49.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:49 smithi037 ceph-mon[29191]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:49.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:49 smithi189 ceph-mon[36179]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:51.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:51 smithi037 ceph-mon[29191]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:51.876 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:51 smithi189 ceph-mon[36179]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:53.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:53 smithi037 ceph-mon[29191]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:53.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:53 smithi189 ceph-mon[36179]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr:Note: switching to '461a84e6fbbf089011eafd13b98169fb9e9b8cc9'. 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr:state without impacting any branches by switching back to a branch. 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-15T22:58:54.364 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr: git switch -c 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr:Or undo this operation with: 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr: git switch - 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-15T22:58:54.365 INFO:tasks.workunit.client.1.smithi189.stderr:HEAD is now at 461a84e6fbb Merge pull request #56863 from cbodley/wip-65467-squid 2024-04-15T22:58:54.368 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-15T22:58:54.430 INFO:tasks.workunit.client.1.smithi189.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-15T22:58:54.433 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-15T22:58:54.433 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-15T22:58:55.184 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-15T22:58:55.252 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-15T22:58:55.294 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-15T22:58:55.296 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-15T22:58:55.296 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-15T22:58:55.337 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-15T22:58:55.339 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-15T22:58:55.339 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-15T22:58:55.395 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-15T22:58:55.399 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-15T22:58:55.399 DEBUG:teuthology.orchestra.run.smithi189:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=461a84e6fbbf089011eafd13b98169fb9e9b8cc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-15T22:58:55.458 INFO:tasks.workunit.client.1.smithi189.stderr:+ mkdir -p fsstress 2024-04-15T22:58:55.459 INFO:tasks.workunit.client.1.smithi189.stderr:+ pushd fsstress 2024-04-15T22:58:55.459 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-15T22:58:55.460 INFO:tasks.workunit.client.1.smithi189.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-15T22:58:55.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:55 smithi037 ceph-mon[29191]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:55.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:55 smithi189 ceph-mon[36179]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:56.238 INFO:tasks.workunit.client.0.smithi037.stderr:Updating files: 99% (12388/12482) Updating files: 100% (12482/12482) Updating files: 100% (12482/12482), done. 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr:Note: switching to '461a84e6fbbf089011eafd13b98169fb9e9b8cc9'. 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr:state without impacting any branches by switching back to a branch. 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr: git switch -c 2024-04-15T22:58:56.672 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr:Or undo this operation with: 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr: git switch - 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-04-15T22:58:56.673 INFO:tasks.workunit.client.0.smithi037.stderr:HEAD is now at 461a84e6fbb Merge pull request #56863 from cbodley/wip-65467-squid 2024-04-15T22:58:56.675 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-15T22:58:56.716 INFO:tasks.workunit.client.0.smithi037.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-15T22:58:56.718 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-15T22:58:56.718 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-15T22:58:57.568 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-15T22:58:57.641 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-15T22:58:57.682 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-15T22:58:57.684 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-15T22:58:57.684 INFO:tasks.workunit.client.0.smithi037.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-15T22:58:57.724 INFO:tasks.workunit.client.0.smithi037.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-15T22:58:57.727 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:58:57.727 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-15T22:58:57.784 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-15T22:58:57.785 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-15T22:58:57.785 DEBUG:teuthology.orchestra.run.smithi037:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=461a84e6fbbf089011eafd13b98169fb9e9b8cc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-15T22:58:57.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:57 smithi037 ceph-mon[29191]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:57.835 INFO:tasks.workunit.client.0.smithi037.stderr:+ mkdir -p fsstress 2024-04-15T22:58:57.836 INFO:tasks.workunit.client.0.smithi037.stderr:+ pushd fsstress 2024-04-15T22:58:57.836 INFO:tasks.workunit.client.0.smithi037.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-15T22:58:57.837 INFO:tasks.workunit.client.0.smithi037.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-15T22:58:57.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:57 smithi189 ceph-mon[36179]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:58.758 INFO:tasks.workunit.client.1.smithi189.stderr:+ tar xzf ltp-full.tgz 2024-04-15T22:58:59.135 INFO:tasks.workunit.client.0.smithi037.stderr:+ tar xzf ltp-full.tgz 2024-04-15T22:58:59.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:58:59 smithi037 ceph-mon[29191]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:58:59.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:58:59 smithi189 ceph-mon[36179]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:59:01.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:01 smithi037 ceph-mon[29191]: pgmap v236: 65 pgs: 65 active+clean; 1.4 MiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s wr, 0 op/s 2024-04-15T22:59:01.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:01 smithi189 ceph-mon[36179]: pgmap v236: 65 pgs: 65 active+clean; 1.4 MiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s wr, 0 op/s 2024-04-15T22:59:03.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:03 smithi189 ceph-mon[36179]: pgmap v237: 65 pgs: 65 active+clean; 1.9 MiB data, 166 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s wr, 12 op/s 2024-04-15T22:59:03.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:59:03.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:59:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:59:03.712 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:03 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:59:03.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:03 smithi037 ceph-mon[29191]: pgmap v237: 65 pgs: 65 active+clean; 1.9 MiB data, 166 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s wr, 12 op/s 2024-04-15T22:59:03.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:59:03.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:59:03.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:59:03.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:03 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' 2024-04-15T22:59:04.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:04 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:59:04.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:04 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:59:05.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:05 smithi189 ceph-mon[36179]: pgmap v238: 65 pgs: 65 active+clean; 13 MiB data, 203 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s wr, 77 op/s 2024-04-15T22:59:05.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:05 smithi037 ceph-mon[29191]: pgmap v238: 65 pgs: 65 active+clean; 13 MiB data, 203 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s wr, 77 op/s 2024-04-15T22:59:07.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:07 smithi037 ceph-mon[29191]: pgmap v239: 65 pgs: 65 active+clean; 39 MiB data, 304 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s wr, 231 op/s 2024-04-15T22:59:07.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:07 smithi189 ceph-mon[36179]: pgmap v239: 65 pgs: 65 active+clean; 39 MiB data, 304 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s wr, 231 op/s 2024-04-15T22:59:09.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:09 smithi037 ceph-mon[29191]: pgmap v240: 65 pgs: 65 active+clean; 49 MiB data, 340 MiB used, 536 GiB / 536 GiB avail; 4.2 MiB/s wr, 298 op/s 2024-04-15T22:59:09.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:09 smithi189 ceph-mon[36179]: pgmap v240: 65 pgs: 65 active+clean; 49 MiB data, 340 MiB used, 536 GiB / 536 GiB avail; 4.2 MiB/s wr, 298 op/s 2024-04-15T22:59:10.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:10 smithi037 ceph-mon[29191]: fsmap cephfs:2 {0=cephfs.smithi037.ifpdxj=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:59:10.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:10 smithi189 ceph-mon[36179]: fsmap cephfs:2 {0=cephfs.smithi037.ifpdxj=up:active,1=cephfs.smithi189.pqjbmb=up:active} 2 up:standby 2024-04-15T22:59:11.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:11 smithi037 ceph-mon[29191]: pgmap v241: 65 pgs: 65 active+clean; 61 MiB data, 354 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 310 op/s 2024-04-15T22:59:11.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:11 smithi189 ceph-mon[36179]: pgmap v241: 65 pgs: 65 active+clean; 61 MiB data, 354 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 310 op/s 2024-04-15T22:59:13.564 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:13 smithi037 ceph-mon[29191]: pgmap v242: 65 pgs: 65 active+clean; 61 MiB data, 356 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 309 op/s 2024-04-15T22:59:13.711 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:13 smithi189 ceph-mon[36179]: pgmap v242: 65 pgs: 65 active+clean; 61 MiB data, 356 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 309 op/s 2024-04-15T22:59:15.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:15 smithi037 ceph-mon[29191]: pgmap v243: 65 pgs: 65 active+clean; 61 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 297 op/s 2024-04-15T22:59:15.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:15 smithi189 ceph-mon[36179]: pgmap v243: 65 pgs: 65 active+clean; 61 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 297 op/s 2024-04-15T22:59:16.966 INFO:tasks.workunit.client.1.smithi189.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-15T22:59:16.967 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-15T22:59:16.967 INFO:tasks.workunit.client.1.smithi189.stderr:+ make 2024-04-15T22:59:17.051 INFO:tasks.workunit.client.1.smithi189.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-15T22:59:17.492 INFO:tasks.workunit.client.1.smithi189.stderr:++ readlink -f fsstress 2024-04-15T22:59:17.493 INFO:tasks.workunit.client.1.smithi189.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-15T22:59:17.493 INFO:tasks.workunit.client.1.smithi189.stderr:+ popd 2024-04-15T22:59:17.494 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-15T22:59:17.494 INFO:tasks.workunit.client.1.smithi189.stderr:+ popd 2024-04-15T22:59:17.494 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-15T22:59:17.494 INFO:tasks.workunit.client.1.smithi189.stderr:++ mktemp -d -p . 2024-04-15T22:59:17.495 INFO:tasks.workunit.client.1.smithi189.stderr:+ T=./tmp.TQzXBHwLQM 2024-04-15T22:59:17.495 INFO:tasks.workunit.client.1.smithi189.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.TQzXBHwLQM -l 1 -n 1000 -p 10 -v 2024-04-15T22:59:17.499 INFO:tasks.workunit.client.1.smithi189.stderr:/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh: line 16: /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress: cannot execute binary file: Exec format error 2024-04-15T22:59:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 126 2024-04-15T22:59:17.500 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-15T22:59:17.500 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-15T22:59:17.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:17 smithi037 ceph-mon[29191]: pgmap v244: 65 pgs: 65 active+clean; 88 MiB data, 493 MiB used, 536 GiB / 536 GiB avail; 6.6 MiB/s wr, 450 op/s 2024-04-15T22:59:17.879 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:17 smithi189 ceph-mon[36179]: pgmap v244: 65 pgs: 65 active+clean; 88 MiB data, 493 MiB used, 536 GiB / 536 GiB avail; 6.6 MiB/s wr, 450 op/s 2024-04-15T22:59:17.881 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi189 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=461a84e6fbbf089011eafd13b98169fb9e9b8cc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' 2024-04-15T22:59:18.107 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=75dd15f32ab747bcb0e47b9418dc6def Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi189 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=461a84e6fbbf089011eafd13b98169fb9e9b8cc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' 2024-04-15T22:59:18.155 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-15T22:59:18.176 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-15T22:59:18.177 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:59:18.177 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T22:59:18.210 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-15T22:59:18.210 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T22:59:18.276 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T22:59:18.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.526+0000 7f6a30789640 1 -- 172.21.15.37:0/697417112 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 msgr2=0x7f6a281076d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:18.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.526+0000 7f6a30789640 1 --2- 172.21.15.37:0/697417112 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 0x7f6a281076d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009370 tx=0x7f6a2002cf00 comp rx=0 tx=0).stop 2024-04-15T22:59:18.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.527+0000 7f6a30789640 1 -- 172.21.15.37:0/697417112 shutdown_connections 2024-04-15T22:59:18.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.527+0000 7f6a30789640 1 --2- 172.21.15.37:0/697417112 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 0x7f6a281076d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.527+0000 7f6a30789640 1 --2- 172.21.15.37:0/697417112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a280716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.527+0000 7f6a30789640 1 -- 172.21.15.37:0/697417112 >> 172.21.15.37:0/697417112 conn(0x7f6a2806c800 msgr2=0x7f6a2806ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:18.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.527+0000 7f6a30789640 1 -- 172.21.15.37:0/697417112 shutdown_connections 2024-04-15T22:59:18.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.527+0000 7f6a30789640 1 -- 172.21.15.37:0/697417112 wait complete. 2024-04-15T22:59:18.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.528+0000 7f6a30789640 1 Processor -- start 2024-04-15T22:59:18.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.528+0000 7f6a30789640 1 -- start start 2024-04-15T22:59:18.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a30789640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a281978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:18.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a30789640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 0x7f6a28197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:18.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a30789640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a281943a0 con 0x7f6a28107260 2024-04-15T22:59:18.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a30789640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a28194510 con 0x7f6a280712a0 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a2e525640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a281978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a2e525640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a281978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.37:54100/0 (socket says 172.21.15.37:54100) 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.530+0000 7f6a2e525640 1 -- 172.21.15.37:0/2278558222 learned_addr learned my addr 172.21.15.37:0/2278558222 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.531+0000 7f6a2dd24640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 0x7f6a28197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.531+0000 7f6a2e525640 1 -- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 msgr2=0x7f6a28197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.531+0000 7f6a2e525640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 0x7f6a28197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.531+0000 7f6a2e525640 1 -- 172.21.15.37:0/2278558222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20009df0 con 0x7f6a280712a0 2024-04-15T22:59:18.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.533+0000 7f6a2e525640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a281978a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00c8f0 tx=0x7f6a1c00cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:18.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.534+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a1c007b80 con 0x7f6a280712a0 2024-04-15T22:59:18.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.534+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6a1c007ce0 con 0x7f6a280712a0 2024-04-15T22:59:18.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.534+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a1c0187d0 con 0x7f6a280712a0 2024-04-15T22:59:18.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.534+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a281947f0 con 0x7f6a280712a0 2024-04-15T22:59:18.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.534+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a28194dd0 con 0x7f6a280712a0 2024-04-15T22:59:18.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.537+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2810e940 con 0x7f6a280712a0 2024-04-15T22:59:18.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.541+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f6a1c018930 con 0x7f6a280712a0 2024-04-15T22:59:18.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.541+0000 7f6a277fe640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6a1006c820 0x7f6a1006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:18.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.542+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6a1c08b8c0 con 0x7f6a280712a0 2024-04-15T22:59:18.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.542+0000 7f6a2dd24640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6a1006c820 0x7f6a1006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:18.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.544+0000 7f6a2dd24640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6a1006c820 0x7f6a1006ecd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a28195820 tx=0x7f6a20000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:18.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.553+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6a1c055fa0 con 0x7f6a280712a0 2024-04-15T22:59:18.755 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:18 smithi037 ceph-mon[29191]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:59:18.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.927+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a281a9cf0 con 0x7f6a280712a0 2024-04-15T22:59:18.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.930+0000 7f6a277fe640 1 -- 172.21.15.37:0/2278558222 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15106 (secure 0 0 0) 0x7f6a1c0595c0 con 0x7f6a280712a0 2024-04-15T22:59:18.923 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-15T22:59:18.923 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":46,"fsid":"60ff499a-fb7a-11ee-bc8f-c7b262605968","created":"2024-04-15T22:49:47.691565+0000","modified":"2024-04-15T22:57:23.443638+0000","last_up_change":"2024-04-15T22:55:51.865136+0000","last_in_change":"2024-04-15T22:55:32.171674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:54:34.664768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-15T22:57:06.044854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-15T22:57:06.646368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cc1abff5-8aea-47ca-982a-29cd590fc31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6803","nonce":2814326386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6805","nonce":2814326386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6809","nonce":2814326386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":2814326386},{"type":"v1","addr":"172.21.15.37:6807","nonce":2814326386}]},"public_addr":"172.21.15.37:6803/2814326386","cluster_addr":"172.21.15.37:6805/2814326386","heartbeat_back_addr":"172.21.15.37:6809/2814326386","heartbeat_front_addr":"172.21.15.37:6807/2814326386","state":["exists","up"]},{"osd":1,"uuid":"e875b0b4-8220-4f8a-9d7b-c2dffdd988dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6811","nonce":1887622200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6813","nonce":1887622200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6817","nonce":1887622200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":1887622200},{"type":"v1","addr":"172.21.15.37:6815","nonce":1887622200}]},"public_addr":"172.21.15.37:6811/1887622200","cluster_addr":"172.21.15.37:6813/1887622200","heartbeat_back_addr":"172.21.15.37:6817/1887622200","heartbeat_front_addr":"172.21.15.37:6815/1887622200","state":["exists","up"]},{"osd":2,"uuid":"08257888-e768-4845-9efa-8358f6e8caec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6819","nonce":503233976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6821","nonce":503233976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6825","nonce":503233976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":503233976},{"type":"v1","addr":"172.21.15.37:6823","nonce":503233976}]},"public_addr":"172.21.15.37:6819/503233976","cluster_addr":"172.21.15.37:6821/503233976","heartbeat_back_addr":"172.21.15.37:6825/503233976","heartbeat_front_addr":"172.21.15.37:6823/503233976","state":["exists","up"]},{"osd":3,"uuid":"1d49e4a2-0131-436b-b02c-77dd4237bee8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6801","nonce":1840940998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6803","nonce":1840940998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6807","nonce":1840940998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":1840940998},{"type":"v1","addr":"172.21.15.189:6805","nonce":1840940998}]},"public_addr":"172.21.15.189:6801/1840940998","cluster_addr":"172.21.15.189:6803/1840940998","heartbeat_back_addr":"172.21.15.189:6807/1840940998","heartbeat_front_addr":"172.21.15.189:6805/1840940998","state":["exists","up"]},{"osd":4,"uuid":"b93aee26-d0dd-4cbc-b45e-b005033c1a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6809","nonce":2518250570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6811","nonce":2518250570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6815","nonce":2518250570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2518250570},{"type":"v1","addr":"172.21.15.189:6813","nonce":2518250570}]},"public_addr":"172.21.15.189:6809/2518250570","cluster_addr":"172.21.15.189:6811/2518250570","heartbeat_back_addr":"172.21.15.189:6815/2518250570","heartbeat_front_addr":"172.21.15.189:6813/2518250570","state":["exists","up"]},{"osd":5,"uuid":"de638b57-8f16-4a8f-b0b0-bbc7d84831c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6817","nonce":580752685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6819","nonce":580752685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6823","nonce":580752685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":580752685},{"type":"v1","addr":"172.21.15.189:6821","nonce":580752685}]},"public_addr":"172.21.15.189:6817/580752685","cluster_addr":"172.21.15.189:6819/580752685","heartbeat_back_addr":"172.21.15.189:6823/580752685","heartbeat_front_addr":"172.21.15.189:6821/580752685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:53:35.515515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:02.780098+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:31.718613+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:54:56.852313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:23.477486+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:55:50.019389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.37:0/3522001690":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/1091007617":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6800/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3560860596":"2024-04-16T22:50:20.083004+0000","172.21.15.37:6801/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/957260578":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/712008483":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6800/85756762":"2024-04-16T22:50:53.072525+0000","172.21.15.37:6801/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/8957815":"2024-04-16T22:50:53.072525+0000","172.21.15.37:0/2768618180":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/3190201420":"2024-04-16T22:52:18.338468+0000","172.21.15.37:0/3357942121":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6827/479125410":"2024-04-16T22:57:23.443615+0000","172.21.15.37:6800/202293977":"2024-04-16T22:50:20.083004+0000","172.21.15.37:0/1434563686":"2024-04-16T22:52:18.338468+0000","172.21.15.37:6826/479125410":"2024-04-16T22:57:23.443615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:59:18.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6a1006c820 msgr2=0x7f6a1006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:18.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6a1006c820 0x7f6a1006ecd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a28195820 tx=0x7f6a20000bc0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 msgr2=0x7f6a281978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a281978a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00c8f0 tx=0x7f6a1c00cdb0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 shutdown_connections 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f6a1006c820 0x7f6a1006ecd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a28107260 0x7f6a28197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 --2- 172.21.15.37:0/2278558222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a280712a0 0x7f6a281978a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 >> 172.21.15.37:0/2278558222 conn(0x7f6a2806c800 msgr2=0x7f6a2810a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 shutdown_connections 2024-04-15T22:59:18.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:18.933+0000 7f6a30789640 1 -- 172.21.15.37:0/2278558222 wait complete. 2024-04-15T22:59:18.943 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T22:54:34.664768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-15T22:57:06.044854+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-15T22:57:06.646368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T22:59:18.944 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T22:59:18.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:18 smithi189 ceph-mon[36179]: from='mgr.14223 172.21.15.37:0/4123997711' entity='mgr.smithi037.gtwkck' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-15T22:59:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.142+0000 7f7e3820b640 1 -- 172.21.15.37:0/1679550277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e300ff660 msgr2=0x7f7e300ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.142+0000 7f7e3820b640 1 --2- 172.21.15.37:0/1679550277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e300ff660 0x7f7e300ffa70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00dc90 tx=0x7f7e2c02fcd0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.145+0000 7f7e3820b640 1 -- 172.21.15.37:0/1679550277 shutdown_connections 2024-04-15T22:59:19.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.145+0000 7f7e3820b640 1 --2- 172.21.15.37:0/1679550277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e301008f0 0x7f7e30100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.145+0000 7f7e3820b640 1 --2- 172.21.15.37:0/1679550277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e300ff660 0x7f7e300ffa70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.145+0000 7f7e3820b640 1 -- 172.21.15.37:0/1679550277 >> 172.21.15.37:0/1679550277 conn(0x7f7e300fac80 msgr2=0x7f7e300fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:19.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.147+0000 7f7e3820b640 1 -- 172.21.15.37:0/1679550277 shutdown_connections 2024-04-15T22:59:19.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.147+0000 7f7e3820b640 1 -- 172.21.15.37:0/1679550277 wait complete. 2024-04-15T22:59:19.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.147+0000 7f7e3820b640 1 Processor -- start 2024-04-15T22:59:19.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e3820b640 1 -- start start 2024-04-15T22:59:19.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e3820b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 0x7f7e301a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:19.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e3820b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e301008f0 0x7f7e301a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:19.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e3820b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e301a4e70 con 0x7f7e300ff660 2024-04-15T22:59:19.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e3820b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e301a4fe0 con 0x7f7e301008f0 2024-04-15T22:59:19.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e35fa7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 0x7f7e301a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e35fa7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 0x7f7e301a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51940/0 (socket says 172.21.15.37:51940) 2024-04-15T22:59:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.148+0000 7f7e35fa7640 1 -- 172.21.15.37:0/3379778198 learned_addr learned my addr 172.21.15.37:0/3379778198 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:59:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.149+0000 7f7e35fa7640 1 -- 172.21.15.37:0/3379778198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e301008f0 msgr2=0x7f7e301a48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.149+0000 7f7e35fa7640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e301008f0 0x7f7e301a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.149+0000 7f7e35fa7640 1 -- 172.21.15.37:0/3379778198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c00d970 con 0x7f7e300ff660 2024-04-15T22:59:19.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.149+0000 7f7e35fa7640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 0x7f7e301a4360 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c005ab0 tx=0x7f7e2c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:19.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.151+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e2c004020 con 0x7f7e300ff660 2024-04-15T22:59:19.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.151+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e301a9540 con 0x7f7e300ff660 2024-04-15T22:59:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.151+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e301a9ac0 con 0x7f7e300ff660 2024-04-15T22:59:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.151+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7e2c0306f0 con 0x7f7e300ff660 2024-04-15T22:59:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.151+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e2c003ac0 con 0x7f7e300ff660 2024-04-15T22:59:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.155+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f7e2c003c20 con 0x7f7e300ff660 2024-04-15T22:59:19.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.155+0000 7f7e26ffd640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f7e1c06c760 0x7f7e1c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:19.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.156+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7e2c0ad6d0 con 0x7f7e300ff660 2024-04-15T22:59:19.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.156+0000 7f7e357a6640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f7e1c06c760 0x7f7e1c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:19.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.158+0000 7f7e357a6640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f7e1c06c760 0x7f7e1c06ec10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7e301a53a0 tx=0x7f7e20006c90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.160+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e14005280 con 0x7f7e300ff660 2024-04-15T22:59:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.169+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7e2c077db0 con 0x7f7e300ff660 2024-04-15T22:59:19.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.538+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e14005ab0 con 0x7f7e300ff660 2024-04-15T22:59:19.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.543+0000 7f7e26ffd640 1 -- 172.21.15.37:0/3379778198 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e2c07b3d0 con 0x7f7e300ff660 2024-04-15T22:59:19.536 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-04-15T22:59:19.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.546+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f7e1c06c760 msgr2=0x7f7e1c06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:19.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.546+0000 7f7e3820b640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f7e1c06c760 0x7f7e1c06ec10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7e301a53a0 tx=0x7f7e20006c90 comp rx=0 tx=0).stop 2024-04-15T22:59:19.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.546+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 msgr2=0x7f7e301a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:19.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.546+0000 7f7e3820b640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 0x7f7e301a4360 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c005ab0 tx=0x7f7e2c0048f0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.547+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 shutdown_connections 2024-04-15T22:59:19.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.547+0000 7f7e3820b640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f7e1c06c760 0x7f7e1c06ec10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.547+0000 7f7e3820b640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e301008f0 0x7f7e301a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.547+0000 7f7e3820b640 1 --2- 172.21.15.37:0/3379778198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e300ff660 0x7f7e301a4360 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c005ab0 tx=0x7f7e2c0048f0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.547+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 >> 172.21.15.37:0/3379778198 conn(0x7f7e300fac80 msgr2=0x7f7e30103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:19.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.547+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 shutdown_connections 2024-04-15T22:59:19.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.548+0000 7f7e3820b640 1 -- 172.21.15.37:0/3379778198 wait complete. 2024-04-15T22:59:19.558 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-15T22:59:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.760+0000 7f67571ba640 1 -- 172.21.15.37:0/3403837935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fdb10 msgr2=0x7f67500fdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.760+0000 7f67571ba640 1 --2- 172.21.15.37:0/3403837935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fdb10 0x7f67500fdf20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6748009880 tx=0x7f674802cf00 comp rx=0 tx=0).stop 2024-04-15T22:59:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.761+0000 7f67571ba640 1 -- 172.21.15.37:0/3403837935 shutdown_connections 2024-04-15T22:59:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.761+0000 7f67571ba640 1 --2- 172.21.15.37:0/3403837935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fed00 0x7f67500ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.761+0000 7f67571ba640 1 --2- 172.21.15.37:0/3403837935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fdb10 0x7f67500fdf20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.761+0000 7f67571ba640 1 -- 172.21.15.37:0/3403837935 >> 172.21.15.37:0/3403837935 conn(0x7f67500f9070 msgr2=0x7f67500fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.762+0000 7f67571ba640 1 -- 172.21.15.37:0/3403837935 shutdown_connections 2024-04-15T22:59:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.762+0000 7f67571ba640 1 -- 172.21.15.37:0/3403837935 wait complete. 2024-04-15T22:59:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.763+0000 7f67571ba640 1 Processor -- start 2024-04-15T22:59:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.763+0000 7f67571ba640 1 -- start start 2024-04-15T22:59:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.763+0000 7f67571ba640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 0x7f6750069e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.763+0000 7f67571ba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fed00 0x7f675006a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.763+0000 7f67571ba640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675006a9c0 con 0x7f67500fdb10 2024-04-15T22:59:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.763+0000 7f67571ba640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675006ab30 con 0x7f67500fed00 2024-04-15T22:59:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.764+0000 7f67561b8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 0x7f6750069e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.764+0000 7f67561b8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 0x7f6750069e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51852/0 (socket says 172.21.15.37:51852) 2024-04-15T22:59:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.764+0000 7f67561b8640 1 -- 172.21.15.37:0/137741596 learned_addr learned my addr 172.21.15.37:0/137741596 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:59:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.764+0000 7f67561b8640 1 -- 172.21.15.37:0/137741596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fed00 msgr2=0x7f675006a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.764+0000 7f67561b8640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fed00 0x7f675006a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.764+0000 7f67561b8640 1 -- 172.21.15.37:0/137741596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748009560 con 0x7f67500fdb10 2024-04-15T22:59:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.765+0000 7f67561b8640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 0x7f6750069e60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6748009850 tx=0x7f6748004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.765+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674802da70 con 0x7f67500fdb10 2024-04-15T22:59:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.765+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f674802dbd0 con 0x7f67500fdb10 2024-04-15T22:59:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.765+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6748035670 con 0x7f67500fdb10 2024-04-15T22:59:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.766+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675006f490 con 0x7f67500fdb10 2024-04-15T22:59:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.766+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675006fa10 con 0x7f67500fdb10 2024-04-15T22:59:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.768+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6750104ca0 con 0x7f67500fdb10 2024-04-15T22:59:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.770+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f67480035f0 con 0x7f67500fdb10 2024-04-15T22:59:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.770+0000 7f674f7fe640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f673806c760 0x7f673806ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.770+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f67480ad620 con 0x7f67500fdb10 2024-04-15T22:59:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.771+0000 7f67559b7640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f673806c760 0x7f673806ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.771+0000 7f67559b7640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f673806c760 0x7f673806ec10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f675006ae90 tx=0x7f6740009250 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:19.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:19.779+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6748077d00 con 0x7f67500fdb10 2024-04-15T22:59:19.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:19 smithi037 ceph-mon[29191]: pgmap v245: 65 pgs: 65 active+clean; 100 MiB data, 518 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 405 op/s 2024-04-15T22:59:19.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:19 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/2278558222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:59:19.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:19 smithi189 ceph-mon[36179]: pgmap v245: 65 pgs: 65 active+clean; 100 MiB data, 518 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 405 op/s 2024-04-15T22:59:19.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:19 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/2278558222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:59:20.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.119+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6750070190 con 0x7f67500fdb10 2024-04-15T22:59:20.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.120+0000 7f674f7fe640 1 -- 172.21.15.37:0/137741596 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f674807b320 con 0x7f67500fdb10 2024-04-15T22:59:20.111 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-04-15T22:59:20.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f673806c760 msgr2=0x7f673806ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f673806c760 0x7f673806ec10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f675006ae90 tx=0x7f6740009250 comp rx=0 tx=0).stop 2024-04-15T22:59:20.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 msgr2=0x7f6750069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 0x7f6750069e60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6748009850 tx=0x7f6748004270 comp rx=0 tx=0).stop 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 shutdown_connections 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f673806c760 0x7f673806ec10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67500fed00 0x7f675006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 --2- 172.21.15.37:0/137741596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67500fdb10 0x7f6750069e60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 >> 172.21.15.37:0/137741596 conn(0x7f67500f9070 msgr2=0x7f6750101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 shutdown_connections 2024-04-15T22:59:20.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.123+0000 7f67571ba640 1 -- 172.21.15.37:0/137741596 wait complete. 2024-04-15T22:59:20.125 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-15T22:59:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 -- 172.21.15.37:0/2254951089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 msgr2=0x7fa2240fe2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/2254951089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa2240fe2c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa214009880 tx=0x7fa21402bce0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 -- 172.21.15.37:0/2254951089 shutdown_connections 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/2254951089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 0x7fa2240fec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/2254951089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa2240fe2c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 -- 172.21.15.37:0/2254951089 >> 172.21.15.37:0/2254951089 conn(0x7fa2240f9a80 msgr2=0x7fa2240fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 -- 172.21.15.37:0/2254951089 shutdown_connections 2024-04-15T22:59:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.268+0000 7fa22a8e0640 1 -- 172.21.15.37:0/2254951089 wait complete. 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa22a8e0640 1 Processor -- start 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa22a8e0640 1 -- start start 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa22a8e0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa224197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa22a8e0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 0x7fa224197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa22a8e0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224197c50 con 0x7fa2240fdeb0 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa22a8e0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224197dc0 con 0x7fa2240fe800 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.269+0000 7fa223fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa224197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:20.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa2237fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 0x7fa224197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa223fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa224197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51868/0 (socket says 172.21.15.37:51868) 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa223fff640 1 -- 172.21.15.37:0/1249476490 learned_addr learned my addr 172.21.15.37:0/1249476490 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa223fff640 1 -- 172.21.15.37:0/1249476490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 msgr2=0x7fa224197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa223fff640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 0x7fa224197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa223fff640 1 -- 172.21.15.37:0/1249476490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214009560 con 0x7fa2240fdeb0 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa2237fe640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 0x7fa224197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa223fff640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa224197140 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa214009230 tx=0x7fa21402cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:20.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa214004420 con 0x7fa2240fdeb0 2024-04-15T22:59:20.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa2140054b0 con 0x7fa2240fdeb0 2024-04-15T22:59:20.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa214034440 con 0x7fa2240fdeb0 2024-04-15T22:59:20.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22419c7f0 con 0x7fa2240fdeb0 2024-04-15T22:59:20.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.270+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22419cca0 con 0x7fa2240fdeb0 2024-04-15T22:59:20.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.272+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22410a570 con 0x7fa2240fdeb0 2024-04-15T22:59:20.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.273+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa2140037d0 con 0x7fa2240fdeb0 2024-04-15T22:59:20.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.273+0000 7fa2217fa640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa20c06c5f0 0x7fa20c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:20.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.273+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa2140add60 con 0x7fa2240fdeb0 2024-04-15T22:59:20.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.273+0000 7fa2237fe640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa20c06c5f0 0x7fa20c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:20.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.274+0000 7fa2237fe640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa20c06c5f0 0x7fa20c06eaa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa224198650 tx=0x7fa218005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:20.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.282+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa214078490 con 0x7fa2240fdeb0 2024-04-15T22:59:20.286 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60ff499a-fb7a-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T22:59:20.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.481+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fa22419d110 con 0x7fa2240fdeb0 2024-04-15T22:59:20.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.482+0000 7fa2217fa640 1 -- 172.21.15.37:0/1249476490 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa21404b020 con 0x7fa2240fdeb0 2024-04-15T22:59:20.474 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 32 2024-04-15T22:59:20.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa20c06c5f0 msgr2=0x7fa20c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa20c06c5f0 0x7fa20c06eaa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa224198650 tx=0x7fa218005f90 comp rx=0 tx=0).stop 2024-04-15T22:59:20.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 msgr2=0x7fa224197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa224197140 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa214009230 tx=0x7fa21402cec0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 shutdown_connections 2024-04-15T22:59:20.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7fa20c06c5f0 0x7fa20c06eaa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2240fe800 0x7fa224197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 --2- 172.21.15.37:0/1249476490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2240fdeb0 0x7fa224197140 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 >> 172.21.15.37:0/1249476490 conn(0x7fa2240f9a80 msgr2=0x7fa2241064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:20.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 shutdown_connections 2024-04-15T22:59:20.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.485+0000 7fa22a8e0640 1 -- 172.21.15.37:0/1249476490 wait complete. 2024-04-15T22:59:20.494 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T22:59:20.579 INFO:teuthology.orchestra.run.smithi037.stderr:Inferring config /var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/mon.smithi037/config 2024-04-15T22:59:20.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 -- 172.21.15.37:0/2181538660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 msgr2=0x7f03440fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 --2- 172.21.15.37:0/2181538660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f03440fe960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0334009880 tx=0x7f033402bce0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 -- 172.21.15.37:0/2181538660 shutdown_connections 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 --2- 172.21.15.37:0/2181538660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03440ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 --2- 172.21.15.37:0/2181538660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f03440fe960 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 -- 172.21.15.37:0/2181538660 >> 172.21.15.37:0/2181538660 conn(0x7f03440f9a90 msgr2=0x7f03440fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 -- 172.21.15.37:0/2181538660 shutdown_connections 2024-04-15T22:59:20.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.640+0000 7f034b9a4640 1 -- 172.21.15.37:0/2181538660 wait complete. 2024-04-15T22:59:20.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.641+0000 7f034b9a4640 1 Processor -- start 2024-04-15T22:59:20.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.641+0000 7f034b9a4640 1 -- start start 2024-04-15T22:59:20.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.641+0000 7f034b9a4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f0344193db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:20.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.641+0000 7f034b9a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03441942f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:20.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.641+0000 7f034b9a4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03441948c0 con 0x7f03440fe550 2024-04-15T22:59:20.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.641+0000 7f034b9a4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0344194a30 con 0x7f03440ff740 2024-04-15T22:59:20.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0349740640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f0344193db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:20.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03441942f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03441942f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.37:38328/0 (socket says 172.21.15.37:38328) 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0349740640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f0344193db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51884/0 (socket says 172.21.15.37:51884) 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 -- 172.21.15.37:0/1056017647 learned_addr learned my addr 172.21.15.37:0/1056017647 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 -- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 msgr2=0x7f0344193db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f0344193db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 -- 172.21.15.37:0/1056017647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334009560 con 0x7f03440ff740 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f0348f3f640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03441942f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f034000eb00 tx=0x7f0340009f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0340013070 con 0x7f03440ff740 2024-04-15T22:59:20.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f034000bd80 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0344067370 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.642+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0340016780 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.643+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03440678a0 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.644+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0344107110 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.644+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0340016900 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.645+0000 7f033a7fc640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f033006c6b0 0x7f033006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.645+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f034008d2b0 con 0x7f03440ff740 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.645+0000 7f0349740640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f033006c6b0 0x7f033006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:20.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.645+0000 7f0349740640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f033006c6b0 0x7f033006eb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0334000c00 tx=0x7f0334005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:20.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.649+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0340051720 con 0x7f03440ff740 2024-04-15T22:59:20.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:20 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/3379778198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:59:20.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:20 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/137741596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-15T22:59:20.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:20 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1249476490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-15T22:59:20.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.881+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0344067e20 con 0x7f03440ff740 2024-04-15T22:59:20.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.881+0000 7f033a7fc640 1 -- 172.21.15.37:0/1056017647 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v46) v1 ==== 81+0+996 (secure 0 0 0) 0x7f0340057a00 con 0x7f03440ff740 2024-04-15T22:59:20.872 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/3522001690 2024-04-16T22:50:53.072525+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/1091007617 2024-04-16T22:50:20.083004+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6800/712008483 2024-04-16T22:52:18.338468+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/3560860596 2024-04-16T22:50:20.083004+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6801/85756762 2024-04-16T22:50:53.072525+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/957260578 2024-04-16T22:50:53.072525+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6801/712008483 2024-04-16T22:52:18.338468+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6800/85756762 2024-04-16T22:50:53.072525+0000 2024-04-15T22:59:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6801/202293977 2024-04-16T22:50:20.083004+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/8957815 2024-04-16T22:50:53.072525+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/2768618180 2024-04-16T22:50:20.083004+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/3190201420 2024-04-16T22:52:18.338468+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/3357942121 2024-04-16T22:52:18.338468+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6827/479125410 2024-04-16T22:57:23.443615+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6800/202293977 2024-04-16T22:50:20.083004+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:0/1434563686 2024-04-16T22:52:18.338468+0000 2024-04-15T22:59:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37:6826/479125410 2024-04-16T22:57:23.443615+0000 2024-04-15T22:59:20.875 INFO:teuthology.orchestra.run.smithi037.stderr:listed 17 entries 2024-04-15T22:59:20.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f033006c6b0 msgr2=0x7f033006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f033006c6b0 0x7f033006eb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0334000c00 tx=0x7f0334005fd0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 msgr2=0x7f03441942f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03441942f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f034000eb00 tx=0x7f0340009f20 comp rx=0 tx=0).stop 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 shutdown_connections 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f033006c6b0 0x7f033006eb60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03440ff740 0x7f03441942f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 --2- 172.21.15.37:0/1056017647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03440fe550 0x7f0344193db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 >> 172.21.15.37:0/1056017647 conn(0x7f03440f9a90 msgr2=0x7f0344102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:20.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 shutdown_connections 2024-04-15T22:59:20.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:20.883+0000 7f034b9a4640 1 -- 172.21.15.37:0/1056017647 wait complete. 2024-04-15T22:59:20.885 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-15T22:59:20.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:59:20.886 DEBUG:teuthology.orchestra.run.smithi037:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-15T22:59:20.915 INFO:teuthology.orchestra.run.smithi037.stderr:umount: /home/ubuntu/cephtest/mnt.0: target is busy. 2024-04-15T22:59:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-15T22:59:20.918 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.0... 2024-04-15T22:59:20.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T22:59:20.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2024-04-15T22:59:20.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:20 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/3379778198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:59:20.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:20 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/137741596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-15T22:59:20.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:20 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1249476490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-15T22:59:21.814 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:21 smithi037 ceph-mon[29191]: pgmap v246: 65 pgs: 65 active+clean; 139 MiB data, 668 MiB used, 536 GiB / 536 GiB avail; 8.0 MiB/s wr, 776 op/s 2024-04-15T22:59:21.815 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:21 smithi037 ceph-mon[29191]: from='client.? 172.21.15.37:0/1056017647' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T22:59:21.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:21 smithi189 ceph-mon[36179]: pgmap v246: 65 pgs: 65 active+clean; 139 MiB data, 668 MiB used, 536 GiB / 536 GiB avail; 8.0 MiB/s wr, 776 op/s 2024-04-15T22:59:21.961 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:21 smithi189 ceph-mon[36179]: from='client.? 172.21.15.37:0/1056017647' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T22:59:22.216 INFO:tasks.workunit.client.0.smithi037.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-15T22:59:22.223 INFO:tasks.workunit.client.0.smithi037.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-15T22:59:22.223 INFO:tasks.workunit.client.0.smithi037.stderr:+ make 2024-04-15T22:59:22.252 INFO:teuthology.orchestra.run.smithi037.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-04-15T22:59:22.252 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.252 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.252 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 597770 1027 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.253 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.254 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-04-15T22:59:22.254 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.254 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.254 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.254 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.254 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-04-15T22:59:22.255 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.256 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.256 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.256 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.256 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.256 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.257 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.257 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.257 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.257 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.257 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.258 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.258 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.258 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.258 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.258 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.259 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 3w CHR 1,11 0t0 10 /dev/kmsg 2024-04-15T22:59:22.259 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 4u a_inode 0,14 0 1057 [eventpoll:8,11,18,19,34,36,42,43,44,45,49,53,54,55,56,57,58,59,60,61,62,72,73,74,75,96,97,98,99,100,110,111...] 2024-04-15T22:59:22.259 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 5u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.259 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 6r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.259 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 7r DIR 0,25 0 1 /sys/fs/cgroup 2024-04-15T22:59:22.260 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 8u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.260 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 9u a_inode 0,14 0 1057 [eventpoll:10,13] 2024-04-15T22:59:22.260 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 10r REG 0,20 0 16061 /proc/1/mountinfo 2024-04-15T22:59:22.261 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 11r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.261 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 12u netlink 0t0 14757 AUDIT 2024-04-15T22:59:22.261 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 13r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.261 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 14r REG 0,20 0 4026532076 /proc/swaps 2024-04-15T22:59:22.261 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 15u netlink 0t0 14749 KOBJECT_UEVENT 2024-04-15T22:59:22.262 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 16u unix 0xffff964f18737800 0t0 14751 /run/systemd/private type=STREAM (LISTEN) 2024-04-15T22:59:22.262 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 17u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.262 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 18r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.262 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 19r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 20u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 21u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 22u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 23u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 24u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 25u unix 0xffff964f06a7b700 0t0 20011 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 26u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 27r CHR 10,235 0t0 111 /dev/autofs 2024-04-15T22:59:22.263 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 28u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 29u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 30u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 31u a_inode 0,14 0 1057 bpf-prog 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 34u unix 0xffff964f03072300 0t0 16063 /run/systemd/notify type=DGRAM (CONNECTED) 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 36u unix 0xffff964f03073c00 0t0 16064 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 37u unix 0xffff964f03077800 0t0 16065 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 42u unix 0xffff964f18736400 0t0 21996 /run/.heim_org.h5l.kcm-socket type=STREAM (LISTEN) 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 43u unix 0xffff964f040af300 0t0 1220 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.264 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 44u unix 0xffff964f03088500 0t0 16069 /run/systemd/io.system.ManagedOOM type=STREAM (LISTEN) 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 45u unix 0xffff964f0308ad00 0t0 16068 /run/systemd/userdb/io.systemd.DynamicUser type=STREAM (LISTEN) 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 49u unix 0xffff964f040ac600 0t0 13823 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 51u unix 0xffff964f040aee00 0t0 20190 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 53u unix 0xffff964f06a68f00 0t0 20552 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 54u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 55u unix 0xffff964f18376900 0t0 17984 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.265 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 56u unix 0xffff964f06a68a00 0t0 17985 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.266 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 57u unix 0xffff964f06a6cb00 0t0 17986 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.266 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 58u unix 0xffff964f18371400 0t0 17989 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.266 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 59u unix 0xffff964f18325000 0t0 20409 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.266 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 60u unix 0xffff964f18735000 0t0 22002 type=STREAM (CONNECTED) 2024-04-15T22:59:22.266 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 61u unix 0xffff964f06a6e400 0t0 18070 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.266 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 62u unix 0xffff964f040ad000 0t0 18071 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 72u unix 0xffff964f18326900 0t0 18872 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 73u unix 0xffff964f18732800 0t0 17115 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 74u unix 0xffff964f2229cb00 0t0 86973 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 75u unix 0xffff964f18675500 0t0 118518 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 85u unix 0xffff964f18733c00 0t0 21994 /run/dbus/system_bus_socket type=STREAM (LISTEN) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 86u unix 0xffff964f03074600 0t0 16078 /run/systemd/journal/socket type=DGRAM (CONNECTED) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 87u unix 0xffff964f0308ee00 0t0 16080 /run/systemd/journal/stdout type=STREAM (LISTEN) 2024-04-15T22:59:22.267 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 96u CHR 10,242 0t0 165 /dev/rfkill 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 97r FIFO 0,13 0t0 19958 pipe 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 98u FIFO 0,24 0t0 447 /run/dmeventd-server 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 99u FIFO 0,24 0t0 448 /run/dmeventd-client 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 100u unix 0xffff964f06a6f800 0t0 19961 /run/lvm/lvmpolld.socket type=STREAM (LISTEN) 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 101u netlink 0t0 19977 KOBJECT_UEVENT 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 102u unix 0xffff964f03075f00 0t0 16076 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 109u unix 0xffff964f06a7f300 0t0 19975 /run/udev/control type=SEQPACKET (LISTEN) 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 110u FIFO 0,24 0t0 453 /run/initctl 2024-04-15T22:59:22.268 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 111u unix 0xffff964f06a7d500 0t0 19973 /run/systemd/coredump type=SEQPACKET (LISTEN) 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 119u unix 0xffff964f06a6b700 0t0 19963 /run/rpcbind.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 120u IPv4 18628 0t0 TCP *:sunrpc (LISTEN) 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 121u IPv4 18634 0t0 UDP *:sunrpc 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 122u IPv6 18640 0t0 TCP *:sunrpc (LISTEN) 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 1 root 123u IPv6 18646 0t0 UDP *:sunrpc 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:kthreadd 2 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:kthreadd 2 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.269 INFO:teuthology.orchestra.run.smithi037.stdout:kthreadd 2 root txt unknown /proc/2/exe 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_gp 3 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_gp 3 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_gp 3 root txt unknown /proc/3/exe 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_par_g 4 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_par_g 4 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_par_g 4 root txt unknown /proc/4/exe 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:slub_flus 5 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.270 INFO:teuthology.orchestra.run.smithi037.stdout:slub_flus 5 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:slub_flus 5 root txt unknown /proc/5/exe 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:netns 6 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:netns 6 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:netns 6 root txt unknown /proc/6/exe 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 8 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 8 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 8 root txt unknown /proc/8/exe 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:mm_percpu 10 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.271 INFO:teuthology.orchestra.run.smithi037.stdout:mm_percpu 10 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:mm_percpu 10 root txt unknown /proc/10/exe 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 12 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 12 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 12 root txt unknown /proc/12/exe 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 13 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 13 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 13 root txt unknown /proc/13/exe 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 14 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.272 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 14 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_tasks 14 root txt unknown /proc/14/exe 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 15 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 15 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 15 root txt unknown /proc/15/exe 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:pr/ttyS1 16 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:pr/ttyS1 16 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:pr/ttyS1 16 root txt unknown /proc/16/exe 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_preem 17 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.273 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_preem 17 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:rcu_preem 17 root txt unknown /proc/17/exe 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:migration 18 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:migration 18 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:migration 18 root txt unknown /proc/18/exe 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 19 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 19 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 19 root txt unknown /proc/19/exe 2024-04-15T22:59:22.274 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/0 21 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/0 21 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/0 21 root txt unknown /proc/21/exe 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/1 22 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/1 22 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/1 22 root txt unknown /proc/22/exe 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 23 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.275 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 23 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.276 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 23 root txt unknown /proc/23/exe 2024-04-15T22:59:22.276 INFO:teuthology.orchestra.run.smithi037.stdout:migration 24 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.276 INFO:teuthology.orchestra.run.smithi037.stdout:migration 24 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.276 INFO:teuthology.orchestra.run.smithi037.stdout:migration 24 root txt unknown /proc/24/exe 2024-04-15T22:59:22.276 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 25 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.276 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 25 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.277 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 25 root txt unknown /proc/25/exe 2024-04-15T22:59:22.277 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 27 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.277 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 27 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.277 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 27 root txt unknown /proc/27/exe 2024-04-15T22:59:22.277 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/2 28 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.278 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/2 28 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.278 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/2 28 root txt unknown /proc/28/exe 2024-04-15T22:59:22.278 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 29 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.278 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 29 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.278 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 29 root txt unknown /proc/29/exe 2024-04-15T22:59:22.279 INFO:teuthology.orchestra.run.smithi037.stdout:migration 30 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.279 INFO:teuthology.orchestra.run.smithi037.stdout:migration 30 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.279 INFO:teuthology.orchestra.run.smithi037.stdout:migration 30 root txt unknown /proc/30/exe 2024-04-15T22:59:22.279 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 31 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.279 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 31 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.279 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 31 root txt unknown /proc/31/exe 2024-04-15T22:59:22.280 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 33 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.280 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 33 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.280 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 33 root txt unknown /proc/33/exe 2024-04-15T22:59:22.280 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/3 34 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.280 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/3 34 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.281 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/3 34 root txt unknown /proc/34/exe 2024-04-15T22:59:22.281 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 35 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.281 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 35 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.281 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 35 root txt unknown /proc/35/exe 2024-04-15T22:59:22.281 INFO:teuthology.orchestra.run.smithi037.stdout:migration 36 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.282 INFO:teuthology.orchestra.run.smithi037.stdout:migration 36 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.282 INFO:teuthology.orchestra.run.smithi037.stdout:migration 36 root txt unknown /proc/36/exe 2024-04-15T22:59:22.282 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 37 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.282 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 37 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.282 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 37 root txt unknown /proc/37/exe 2024-04-15T22:59:22.283 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 39 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.283 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 39 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.283 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 39 root txt unknown /proc/39/exe 2024-04-15T22:59:22.283 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/4 40 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.283 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/4 40 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.283 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/4 40 root txt unknown /proc/40/exe 2024-04-15T22:59:22.284 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 41 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.284 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 41 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.284 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 41 root txt unknown /proc/41/exe 2024-04-15T22:59:22.284 INFO:teuthology.orchestra.run.smithi037.stdout:migration 42 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.284 INFO:teuthology.orchestra.run.smithi037.stdout:migration 42 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.285 INFO:teuthology.orchestra.run.smithi037.stdout:migration 42 root txt unknown /proc/42/exe 2024-04-15T22:59:22.285 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 43 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.285 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 43 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.286 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 43 root txt unknown /proc/43/exe 2024-04-15T22:59:22.286 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 44 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.286 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 44 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.286 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 44 root txt unknown /proc/44/exe 2024-04-15T22:59:22.286 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 45 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.287 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 45 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.287 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 45 root txt unknown /proc/45/exe 2024-04-15T22:59:22.287 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/5 46 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.288 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/5 46 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.288 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/5 46 root txt unknown /proc/46/exe 2024-04-15T22:59:22.288 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 47 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.288 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 47 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.289 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 47 root txt unknown /proc/47/exe 2024-04-15T22:59:22.289 INFO:teuthology.orchestra.run.smithi037.stdout:migration 48 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.289 INFO:teuthology.orchestra.run.smithi037.stdout:migration 48 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.289 INFO:teuthology.orchestra.run.smithi037.stdout:migration 48 root txt unknown /proc/48/exe 2024-04-15T22:59:22.289 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 49 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 49 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 49 root txt unknown /proc/49/exe 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 51 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 51 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 51 root txt unknown /proc/51/exe 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/6 52 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/6 52 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.290 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/6 52 root txt unknown /proc/52/exe 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 53 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 53 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 53 root txt unknown /proc/53/exe 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:migration 54 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:migration 54 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:migration 54 root txt unknown /proc/54/exe 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 55 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.291 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 55 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.292 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 55 root txt unknown /proc/55/exe 2024-04-15T22:59:22.293 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 57 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.293 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 57 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.293 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 57 root txt unknown /proc/57/exe 2024-04-15T22:59:22.293 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/7 58 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.293 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/7 58 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.294 INFO:teuthology.orchestra.run.smithi037.stdout:cpuhp/7 58 root txt unknown /proc/58/exe 2024-04-15T22:59:22.294 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 59 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.294 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 59 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.294 INFO:teuthology.orchestra.run.smithi037.stdout:idle_inje 59 root txt unknown /proc/59/exe 2024-04-15T22:59:22.295 INFO:teuthology.orchestra.run.smithi037.stdout:migration 60 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.295 INFO:teuthology.orchestra.run.smithi037.stdout:migration 60 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.295 INFO:teuthology.orchestra.run.smithi037.stdout:migration 60 root txt unknown /proc/60/exe 2024-04-15T22:59:22.295 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 61 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.296 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 61 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.296 INFO:teuthology.orchestra.run.smithi037.stdout:ksoftirqd 61 root txt unknown /proc/61/exe 2024-04-15T22:59:22.296 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 63 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.296 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 63 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.297 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 63 root txt unknown /proc/63/exe 2024-04-15T22:59:22.297 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 65 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.297 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 65 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.297 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 65 root txt unknown /proc/65/exe 2024-04-15T22:59:22.297 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 66 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.298 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 66 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.298 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 66 root txt unknown /proc/66/exe 2024-04-15T22:59:22.298 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 69 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.299 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 69 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.299 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 69 root txt unknown /proc/69/exe 2024-04-15T22:59:22.299 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 70 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.299 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 70 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.299 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 70 root txt unknown /proc/70/exe 2024-04-15T22:59:22.300 INFO:teuthology.orchestra.run.smithi037.stdout:kdevtmpfs 71 root cwd DIR 0,5 3500 1 / 2024-04-15T22:59:22.300 INFO:teuthology.orchestra.run.smithi037.stdout:kdevtmpfs 71 root rtd DIR 0,5 3500 1 / 2024-04-15T22:59:22.300 INFO:teuthology.orchestra.run.smithi037.stdout:kdevtmpfs 71 root txt unknown /proc/71/exe 2024-04-15T22:59:22.300 INFO:teuthology.orchestra.run.smithi037.stdout:inet_frag 72 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.301 INFO:teuthology.orchestra.run.smithi037.stdout:inet_frag 72 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.301 INFO:teuthology.orchestra.run.smithi037.stdout:inet_frag 72 root txt unknown /proc/72/exe 2024-04-15T22:59:22.301 INFO:teuthology.orchestra.run.smithi037.stdout:kauditd 73 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.301 INFO:teuthology.orchestra.run.smithi037.stdout:kauditd 73 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.302 INFO:teuthology.orchestra.run.smithi037.stdout:kauditd 73 root txt unknown /proc/73/exe 2024-04-15T22:59:22.302 INFO:teuthology.orchestra.run.smithi037.stdout:khungtask 75 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.302 INFO:teuthology.orchestra.run.smithi037.stdout:khungtask 75 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.302 INFO:teuthology.orchestra.run.smithi037.stdout:khungtask 75 root txt unknown /proc/75/exe 2024-04-15T22:59:22.302 INFO:teuthology.orchestra.run.smithi037.stdout:oom_reape 76 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:oom_reape 76 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:oom_reape 76 root txt unknown /proc/76/exe 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:writeback 77 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:writeback 77 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:writeback 77 root txt unknown /proc/77/exe 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:kcompactd 78 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:kcompactd 78 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:kcompactd 78 root txt unknown /proc/78/exe 2024-04-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi037.stdout:ksmd 79 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:ksmd 79 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:ksmd 79 root txt unknown /proc/79/exe 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:khugepage 80 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:khugepage 80 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:khugepage 80 root txt unknown /proc/80/exe 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:cryptd 81 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:cryptd 81 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:cryptd 81 root txt unknown /proc/81/exe 2024-04-15T22:59:22.304 INFO:teuthology.orchestra.run.smithi037.stdout:kintegrit 82 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:kintegrit 82 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:kintegrit 82 root txt unknown /proc/82/exe 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:kblockd 83 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:kblockd 83 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:kblockd 83 root txt unknown /proc/83/exe 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:blkcg_pun 84 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:blkcg_pun 84 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:blkcg_pun 84 root txt unknown /proc/84/exe 2024-04-15T22:59:22.305 INFO:teuthology.orchestra.run.smithi037.stdout:tpm_dev_w 86 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.306 INFO:teuthology.orchestra.run.smithi037.stdout:tpm_dev_w 86 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.306 INFO:teuthology.orchestra.run.smithi037.stdout:tpm_dev_w 86 root txt unknown /proc/86/exe 2024-04-15T22:59:22.306 INFO:teuthology.orchestra.run.smithi037.stdout:md 87 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.306 INFO:teuthology.orchestra.run.smithi037.stdout:md 87 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.306 INFO:teuthology.orchestra.run.smithi037.stdout:md 87 root txt unknown /proc/87/exe 2024-04-15T22:59:22.306 INFO:teuthology.orchestra.run.smithi037.stdout:md_bitmap 88 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.307 INFO:teuthology.orchestra.run.smithi037.stdout:md_bitmap 88 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.307 INFO:teuthology.orchestra.run.smithi037.stdout:md_bitmap 88 root txt unknown /proc/88/exe 2024-04-15T22:59:22.307 INFO:teuthology.orchestra.run.smithi037.stdout:edac-poll 89 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.308 INFO:teuthology.orchestra.run.smithi037.stdout:edac-poll 89 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.308 INFO:teuthology.orchestra.run.smithi037.stdout:edac-poll 89 root txt unknown /proc/89/exe 2024-04-15T22:59:22.308 INFO:teuthology.orchestra.run.smithi037.stdout:watchdogd 90 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.308 INFO:teuthology.orchestra.run.smithi037.stdout:watchdogd 90 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.308 INFO:teuthology.orchestra.run.smithi037.stdout:watchdogd 90 root txt unknown /proc/90/exe 2024-04-15T22:59:22.308 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 91 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.309 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 91 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.309 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 91 root txt unknown /proc/91/exe 2024-04-15T22:59:22.309 INFO:teuthology.orchestra.run.smithi037.stdout:kswapd0 92 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.309 INFO:teuthology.orchestra.run.smithi037.stdout:kswapd0 92 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.309 INFO:teuthology.orchestra.run.smithi037.stdout:kswapd0 92 root txt unknown /proc/92/exe 2024-04-15T22:59:22.310 INFO:teuthology.orchestra.run.smithi037.stdout:kthrotld 98 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.310 INFO:teuthology.orchestra.run.smithi037.stdout:kthrotld 98 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.310 INFO:teuthology.orchestra.run.smithi037.stdout:kthrotld 98 root txt unknown /proc/98/exe 2024-04-15T22:59:22.310 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 108 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.310 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 108 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.311 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 108 root txt unknown /proc/108/exe 2024-04-15T22:59:22.311 INFO:teuthology.orchestra.run.smithi037.stdout:acpi_ther 109 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.311 INFO:teuthology.orchestra.run.smithi037.stdout:acpi_ther 109 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.311 INFO:teuthology.orchestra.run.smithi037.stdout:acpi_ther 109 root txt unknown /proc/109/exe 2024-04-15T22:59:22.311 INFO:teuthology.orchestra.run.smithi037.stdout:kmpath_rd 110 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.312 INFO:teuthology.orchestra.run.smithi037.stdout:kmpath_rd 110 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.312 INFO:teuthology.orchestra.run.smithi037.stdout:kmpath_rd 110 root txt unknown /proc/110/exe 2024-04-15T22:59:22.312 INFO:teuthology.orchestra.run.smithi037.stdout:kaluad 111 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.312 INFO:teuthology.orchestra.run.smithi037.stdout:kaluad 111 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.312 INFO:teuthology.orchestra.run.smithi037.stdout:kaluad 111 root txt unknown /proc/111/exe 2024-04-15T22:59:22.313 INFO:teuthology.orchestra.run.smithi037.stdout:mld 112 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.313 INFO:teuthology.orchestra.run.smithi037.stdout:mld 112 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.313 INFO:teuthology.orchestra.run.smithi037.stdout:mld 112 root txt unknown /proc/112/exe 2024-04-15T22:59:22.313 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 113 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.314 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 113 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.314 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 113 root txt unknown /proc/113/exe 2024-04-15T22:59:22.314 INFO:teuthology.orchestra.run.smithi037.stdout:ipv6_addr 114 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.314 INFO:teuthology.orchestra.run.smithi037.stdout:ipv6_addr 114 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.315 INFO:teuthology.orchestra.run.smithi037.stdout:ipv6_addr 114 root txt unknown /proc/114/exe 2024-04-15T22:59:22.315 INFO:teuthology.orchestra.run.smithi037.stdout:kstrp 123 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.315 INFO:teuthology.orchestra.run.smithi037.stdout:kstrp 123 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.315 INFO:teuthology.orchestra.run.smithi037.stdout:kstrp 123 root txt unknown /proc/123/exe 2024-04-15T22:59:22.316 INFO:teuthology.orchestra.run.smithi037.stdout:zswap-shr 131 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.316 INFO:teuthology.orchestra.run.smithi037.stdout:zswap-shr 131 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.316 INFO:teuthology.orchestra.run.smithi037.stdout:zswap-shr 131 root txt unknown /proc/131/exe 2024-04-15T22:59:22.316 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 132 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.317 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 132 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.317 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 132 root txt unknown /proc/132/exe 2024-04-15T22:59:22.318 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 277 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.318 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 277 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.318 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 277 root txt unknown /proc/277/exe 2024-04-15T22:59:22.319 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 281 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.319 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 281 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.319 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/7 281 root txt unknown /proc/281/exe 2024-04-15T22:59:22.319 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 284 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.320 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 284 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.320 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 284 root txt unknown /proc/284/exe 2024-04-15T22:59:22.320 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 285 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.320 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 285 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.321 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 285 root txt unknown /proc/285/exe 2024-04-15T22:59:22.321 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 294 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.321 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 294 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.321 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 294 root txt unknown /proc/294/exe 2024-04-15T22:59:22.322 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 297 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.322 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 297 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.322 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 297 root txt unknown /proc/297/exe 2024-04-15T22:59:22.322 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 347 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.323 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 347 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.323 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 347 root txt unknown /proc/347/exe 2024-04-15T22:59:22.323 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 362 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.323 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 362 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.324 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/5 362 root txt unknown /proc/362/exe 2024-04-15T22:59:22.324 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 386 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.324 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 386 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.324 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 386 root txt unknown /proc/386/exe 2024-04-15T22:59:22.324 INFO:teuthology.orchestra.run.smithi037.stdout:ata_sff 420 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.325 INFO:teuthology.orchestra.run.smithi037.stdout:ata_sff 420 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.325 INFO:teuthology.orchestra.run.smithi037.stdout:ata_sff 420 root txt unknown /proc/420/exe 2024-04-15T22:59:22.325 INFO:teuthology.orchestra.run.smithi037.stdout:ixgbe 423 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.325 INFO:teuthology.orchestra.run.smithi037.stdout:ixgbe 423 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.326 INFO:teuthology.orchestra.run.smithi037.stdout:ixgbe 423 root txt unknown /proc/423/exe 2024-04-15T22:59:22.326 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-wq 429 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.326 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-wq 429 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.326 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-wq 429 root txt unknown /proc/429/exe 2024-04-15T22:59:22.326 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-rese 431 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.327 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-rese 431 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.327 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-rese 431 root txt unknown /proc/431/exe 2024-04-15T22:59:22.327 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-dele 442 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.327 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-dele 442 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.327 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-dele 442 root txt unknown /proc/442/exe 2024-04-15T22:59:22.328 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-auth 445 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.328 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-auth 445 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.328 INFO:teuthology.orchestra.run.smithi037.stdout:nvme-auth 445 root txt unknown /proc/445/exe 2024-04-15T22:59:22.328 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_0 449 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.328 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_0 449 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.329 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_0 449 root txt unknown /proc/449/exe 2024-04-15T22:59:22.329 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 450 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.329 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 450 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.329 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 450 root txt unknown /proc/450/exe 2024-04-15T22:59:22.330 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_1 451 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.330 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_1 451 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.330 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_1 451 root txt unknown /proc/451/exe 2024-04-15T22:59:22.333 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 452 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.334 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 452 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.335 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 452 root txt unknown /proc/452/exe 2024-04-15T22:59:22.335 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_2 453 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.335 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_2 453 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.336 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_2 453 root txt unknown /proc/453/exe 2024-04-15T22:59:22.336 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 454 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.337 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 454 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.337 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 454 root txt unknown /proc/454/exe 2024-04-15T22:59:22.338 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_3 455 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.338 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_3 455 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.339 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_3 455 root txt unknown /proc/455/exe 2024-04-15T22:59:22.339 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 456 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.339 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 456 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.341 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 456 root txt unknown /proc/456/exe 2024-04-15T22:59:22.341 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_4 459 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.342 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_4 459 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.342 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_4 459 root txt unknown /proc/459/exe 2024-04-15T22:59:22.342 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 460 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.344 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 460 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.344 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 460 root txt unknown /proc/460/exe 2024-04-15T22:59:22.344 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_5 461 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.345 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_5 461 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.345 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_5 461 root txt unknown /proc/461/exe 2024-04-15T22:59:22.345 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 462 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.346 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 462 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.346 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 462 root txt unknown /proc/462/exe 2024-04-15T22:59:22.346 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_6 463 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.347 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_6 463 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.347 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_6 463 root txt unknown /proc/463/exe 2024-04-15T22:59:22.347 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 464 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.348 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 464 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.348 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 464 root txt unknown /proc/464/exe 2024-04-15T22:59:22.348 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_7 465 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.349 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_7 465 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.349 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_7 465 root txt unknown /proc/465/exe 2024-04-15T22:59:22.350 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 466 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.350 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 466 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 466 root txt unknown /proc/466/exe 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_8 467 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_8 467 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_8 467 root txt unknown /proc/467/exe 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 468 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 468 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 468 root txt unknown /proc/468/exe 2024-04-15T22:59:22.351 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_9 469 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_9 469 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_eh_9 469 root txt unknown /proc/469/exe 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 470 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 470 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:scsi_tmf_ 470 root txt unknown /proc/470/exe 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 472 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 472 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 472 root txt unknown /proc/472/exe 2024-04-15T22:59:22.352 INFO:teuthology.orchestra.run.smithi037.stdout:jbd2/sda1 502 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:jbd2/sda1 502 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:jbd2/sda1 502 root txt unknown /proc/502/exe 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:ext4-rsv- 503 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:ext4-rsv- 503 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:ext4-rsv- 503 root txt unknown /proc/503/exe 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root txt REG 8,1 185496 20360 /usr/lib/systemd/systemd-journald 2024-04-15T22:59:22.353 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.354 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.355 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root mem REG 0,24 8 55 /run/systemd/journal/kernel-seqnum 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 3u unix 0xffff964f03074600 0t0 16078 /run/systemd/journal/socket type=DGRAM (CONNECTED) 2024-04-15T22:59:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 4u unix 0xffff964f0308ee00 0t0 16080 /run/systemd/journal/stdout type=STREAM (LISTEN) 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 5u unix 0xffff964f03075f00 0t0 16076 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 6w CHR 1,11 0t0 10 /dev/kmsg 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 7u a_inode 0,14 0 1057 [eventpoll:3,4,5,8,9,10,11,12,13,15,17,18,19,20,21,22,23,24,25,27,28,29,30,32,33] 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 8u CHR 1,11 0t0 10 /dev/kmsg 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 9u unix 0xffff964f18733700 0t0 14762 /run/systemd/journal/io.systemd.journal type=STREAM (LISTEN) 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 10r REG 0,20 0 16778 /proc/sys/kernel/hostname 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 11u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 12u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.358 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 13u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 14u unix 0xffff964f1873cb00 0t0 14763 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 15u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 16u REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 17u unix 0xffff964f040af300 0t0 1220 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 18u unix 0xffff964f040aee00 0t0 20190 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 19u unix 0xffff964f06a68f00 0t0 20552 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 20u unix 0xffff964f040ac600 0t0 13823 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.359 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 21u unix 0xffff964f2229cb00 0t0 86973 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 22u unix 0xffff964f18675500 0t0 118518 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 23u unix 0xffff964f18376900 0t0 17984 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 24u unix 0xffff964f06a68a00 0t0 17985 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 25u unix 0xffff964f06a6cb00 0t0 17986 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 27u unix 0xffff964f18325000 0t0 20409 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 28u unix 0xffff964f18371400 0t0 17989 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 29u unix 0xffff964f06a6e400 0t0 18070 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 30u unix 0xffff964f040ad000 0t0 18071 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.360 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 31u REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 32u unix 0xffff964f18326900 0t0 18872 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-j 578 root 33u unix 0xffff964f18732800 0t0 17115 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:ext4lazyi 584 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:ext4lazyi 584 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:ext4lazyi 584 root txt unknown /proc/584/exe 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 588 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 588 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 588 root txt unknown /proc/588/exe 2024-04-15T22:59:22.361 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 590 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 590 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/0 590 root txt unknown /proc/590/exe 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root txt REG 8,1 597744 11595 /usr/bin/udevadm 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 11730661 64128 /etc/udev/hwdb.bin 2024-04-15T22:59:22.362 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 597770 1027 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 587295 265301 /usr/lib/modules/5.14.0-438.el9.x86_64/modules.symbols.bin 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 830988 265298 /usr/lib/modules/5.14.0-438.el9.x86_64/modules.alias.bin 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 399220 265307 /usr/lib/modules/5.14.0-438.el9.x86_64/modules.dep.bin 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.363 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.364 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 11519 265302 /usr/lib/modules/5.14.0-438.el9.x86_64/modules.builtin.bin 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.366 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 1u unix 0xffff964f040ae900 0t0 18683 type=STREAM (CONNECTED) 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 2u unix 0xffff964f040ae900 0t0 18683 type=STREAM (CONNECTED) 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 3u netlink 0t0 19977 KOBJECT_UEVENT 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 4u unix 0xffff964f06a7f300 0t0 19975 /run/udev/control type=SEQPACKET (LISTEN) 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 5u unix 0xffff964f18393700 0t0 18695 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 6u unix 0xffff964f18392800 0t0 18699 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 7u unix 0xffff964f18390a00 0t0 18700 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.367 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 9u a_inode 0,14 0 1057 [eventpoll:3,4,6,8,10,11,14] 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 10u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 11u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 12r REG 8,1 11730661 64128 /etc/udev/hwdb.bin 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-u 598 root 14u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:ipmi-msgh 641 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:ipmi-msgh 641 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:ipmi-msgh 641 root txt unknown /proc/641/exe 2024-04-15T22:59:22.368 INFO:teuthology.orchestra.run.smithi037.stdout:kipmi0 660 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:kipmi0 660 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:kipmi0 660 root txt unknown /proc/660/exe 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc txt REG 8,1 61200 14999 /usr/bin/rpcbind 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.369 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.370 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 1u unix 0xffff964f040a8000 0t0 18766 type=STREAM (CONNECTED) 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 2u unix 0xffff964f040a8000 0t0 18766 type=STREAM (CONNECTED) 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 3u unix 0xffff964f06a6b700 0t0 19963 /run/rpcbind.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 4u IPv4 18628 0t0 TCP *:sunrpc (LISTEN) 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 5u IPv4 18634 0t0 UDP *:sunrpc 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 6u IPv6 18640 0t0 TCP *:sunrpc (LISTEN) 2024-04-15T22:59:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 7u IPv6 18646 0t0 UDP *:sunrpc 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:rpcbind 671 rpc 8rW REG 0,24 0 1188 /run/rpcbind/rpcbind.lock 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:rpciod 676 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:rpciod 676 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:rpciod 676 root txt unknown /proc/676/exe 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:xprtiod 677 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:xprtiod 677 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:xprtiod 677 root txt unknown /proc/677/exe 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.373 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root txt REG 8,1 144576 11776 /usr/sbin/auditd 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.375 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.375 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.399 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.399 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.399 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.399 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 3u netlink 0t0 20516 AUDIT 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 4w REG 8,1 3314336 64145 /var/log/audit/audit.log 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 6u unix 0xffff964f18378500 0t0 20519 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.400 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 7u unix 0xffff964f18376400 0t0 20518 type=STREAM (CONNECTED) 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 8u unix 0xffff964f18373c00 0t0 20520 type=STREAM (CONNECTED) 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 9u unix 0xffff964f18372800 0t0 20521 type=STREAM (CONNECTED) 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 root 10u a_inode 0,14 0 1057 [eventfd:0] 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root txt REG 8,1 144576 11776 /usr/sbin/auditd 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.401 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.402 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 3u netlink 0t0 20516 AUDIT 2024-04-15T22:59:22.403 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 4w REG 8,1 3314336 64145 /var/log/audit/audit.log 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 6u unix 0xffff964f18378500 0t0 20519 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 7u unix 0xffff964f18376400 0t0 20518 type=STREAM (CONNECTED) 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 8u unix 0xffff964f18373c00 0t0 20520 type=STREAM (CONNECTED) 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 9u unix 0xffff964f18372800 0t0 20521 type=STREAM (CONNECTED) 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 681 auditd root 10u a_inode 0,14 0 1057 [eventfd:0] 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root txt REG 8,1 144576 11776 /usr/sbin/auditd 2024-04-15T22:59:22.404 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.405 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 3u netlink 0t0 20516 AUDIT 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 4w REG 8,1 3314336 64145 /var/log/audit/audit.log 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 6u unix 0xffff964f18378500 0t0 20519 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 7u unix 0xffff964f18376400 0t0 20518 type=STREAM (CONNECTED) 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 8u unix 0xffff964f18373c00 0t0 20520 type=STREAM (CONNECTED) 2024-04-15T22:59:22.407 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 9u unix 0xffff964f18372800 0t0 20521 type=STREAM (CONNECTED) 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:auditd 680 683 auditd root 10u a_inode 0,14 0 1057 [eventfd:0] 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root txt REG 8,1 19624 11914 /usr/sbin/sedispatch 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.409 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.410 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root 0u unix 0xffff964f18375000 0t0 20517 type=STREAM (CONNECTED) 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.411 INFO:teuthology.orchestra.run.smithi037.stdout:sedispatc 682 root 3u unix 0xffff964f0308c600 0t0 23424 type=STREAM (CONNECTED) 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus txt REG 8,1 131896 11213 /usr/bin/dbus-broker-launch 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus DEL REG 0,1 2048 /memfd:dbus-broker-log 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.412 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.413 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 1u unix 0xffff964f040ab200 0t0 18779 type=STREAM (CONNECTED) 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 2u unix 0xffff964f040ab200 0t0 18779 type=STREAM (CONNECTED) 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 3u unix 0xffff964f18733c00 0t0 21994 /run/dbus/system_bus_socket type=STREAM (LISTEN) 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 4u unix 0xffff964f18397800 0t0 18789 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 5u a_inode 0,14 0 1057 [eventpoll:6,7,8,9,10,11] 2024-04-15T22:59:22.414 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 6u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 7r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 8u unix 0xffff964f040aa300 0t0 18791 type=STREAM (CONNECTED) 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 9u unix 0xffff964f040aad00 0t0 18793 type=STREAM (CONNECTED) 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 10u a_inode 0,14 0 1057 [pidfd:720] 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 11u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 719 dbus 12u REG 0,1 2097152 2048 /memfd:dbus-broker-log (deleted) 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.415 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus txt REG 8,1 236336 11212 /usr/bin/dbus-broker 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 1u unix 0xffff964f040ab200 0t0 18779 type=STREAM (CONNECTED) 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 2u unix 0xffff964f040ab200 0t0 18779 type=STREAM (CONNECTED) 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 3u sock 0,8 0t0 20547 protocol: NETLINK 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 4u unix 0xffff964f18397800 0t0 18789 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 5u a_inode 0,14 0 1057 [eventpoll:6,7,8,9,10,11,12,13,14,15,18,19,20] 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 6u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.417 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 7u unix 0xffff964f18733c00 0t0 21994 /run/dbus/system_bus_socket type=STREAM (LISTEN) 2024-04-15T22:59:22.418 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 8u unix 0xffff964f18730a00 0t0 20548 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.418 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 9u unix 0xffff964f040a8500 0t0 18792 type=STREAM (CONNECTED) 2024-04-15T22:59:22.418 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 10u unix 0xffff964f040a9900 0t0 20550 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.418 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 11u unix 0xffff964f18370a00 0t0 18794 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.418 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 12u unix 0xffff964f18185a00 0t0 18818 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 13u unix 0xffff964f06a6a300 0t0 20612 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 14u unix 0xffff964f0308cb00 0t0 23425 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 15u unix 0xffff964f040ad500 0t0 20352 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 18u unix 0xffff964f18733200 0t0 22576 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 19u unix 0xffff964f18323c00 0t0 20838 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:dbus-brok 720 dbus 20u unix 0xffff964f03089900 0t0 120924 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-04-15T22:59:22.419 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root txt REG 8,1 3644720 11633 /usr/sbin/NetworkManager 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 317552 52828 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-settings-plugin-ifcfg-rh.so 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 62456 9782 /usr/lib64/libjansson.so.4.14.0 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 31792 10954 /usr/lib64/libteamdctl.so.0.1.5 2024-04-15T22:59:22.420 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 57016 52811 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-device-plugin-team.so 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.421 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 138712 9890 /usr/lib64/libbrotlicommon.so.1.0.9 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.422 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 53048 9892 /usr/lib64/libbrotlidec.so.1.0.9 2024-04-15T22:59:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 77488 9888 /usr/lib64/libpsl.so.5.3.3 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 483672 11481 /usr/lib64/libssh.so.4.9.4 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 169704 11377 /usr/lib64/libnghttp2.so.14.20.1 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.424 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.425 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 662304 11485 /usr/lib64/libcurl.so.4.7.0 2024-04-15T22:59:22.426 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-04-15T22:59:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 32408 11334 /usr/lib64/libndp.so.0.2.0 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.428 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 1u unix 0xffff964f06a6df00 0t0 20197 type=STREAM (CONNECTED) 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 2u unix 0xffff964f06a6df00 0t0 20197 type=STREAM (CONNECTED) 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 3u a_inode 0,14 0 1057 [eventfd:6] 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 4u a_inode 0,14 0 1057 [eventfd:7] 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 5u unix 0xffff964f06a79e00 0t0 20252 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 6u unix 0xffff964f06a6ee00 0t0 20253 type=STREAM (CONNECTED) 2024-04-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 7u a_inode 0,14 0 1057 [eventfd:8] 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 8r REG 0,4 0 4026531840 net 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 9r REG 0,4 0 4026532495 mnt 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 10u netlink 0t0 20257 KOBJECT_UEVENT 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 11u netlink 0t0 20258 GENERIC 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 12u netlink 0t0 20259 GENERIC 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 13u netlink 0t0 20260 ROUTE 2024-04-15T22:59:22.430 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 14r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 15r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 16u netlink 0t0 20614 KOBJECT_UEVENT 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 17u a_inode 0,14 0 1057 [eventpoll] 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 18u unix 0xffff964f040aa800 0t0 18821 type=STREAM (CONNECTED) 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 19u a_inode 0,14 0 1057 [eventpoll:21,27] 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 20w FIFO 0,24 0t0 1306 /run/systemd/inhibit/1.ref 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 21u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.431 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 23u raw6 0t0 18101 00000000000000000000000000000000:003A->00000000000000000000000000000000:0000 st=07 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 root 27u IPv4 18122 0t0 UDP smithi037.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root txt REG 8,1 3644720 11633 /usr/sbin/NetworkManager 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 317552 52828 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-settings-plugin-ifcfg-rh.so 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 62456 9782 /usr/lib64/libjansson.so.4.14.0 2024-04-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 31792 10954 /usr/lib64/libteamdctl.so.0.1.5 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 57016 52811 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-device-plugin-team.so 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 138712 9890 /usr/lib64/libbrotlicommon.so.1.0.9 2024-04-15T22:59:22.433 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.434 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 53048 9892 /usr/lib64/libbrotlidec.so.1.0.9 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-04-15T22:59:22.435 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 77488 9888 /usr/lib64/libpsl.so.5.3.3 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 483672 11481 /usr/lib64/libssh.so.4.9.4 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 169704 11377 /usr/lib64/libnghttp2.so.14.20.1 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-04-15T22:59:22.436 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-04-15T22:59:22.437 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 662304 11485 /usr/lib64/libcurl.so.4.7.0 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-04-15T22:59:22.438 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 32408 11334 /usr/lib64/libndp.so.0.2.0 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.439 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.440 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 1u unix 0xffff964f06a6df00 0t0 20197 type=STREAM (CONNECTED) 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 2u unix 0xffff964f06a6df00 0t0 20197 type=STREAM (CONNECTED) 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 3u a_inode 0,14 0 1057 [eventfd:6] 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 4u a_inode 0,14 0 1057 [eventfd:7] 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 5u unix 0xffff964f06a79e00 0t0 20252 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 6u unix 0xffff964f06a6ee00 0t0 20253 type=STREAM (CONNECTED) 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 7u a_inode 0,14 0 1057 [eventfd:8] 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 8r REG 0,4 0 4026531840 net 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 9r REG 0,4 0 4026532495 mnt 2024-04-15T22:59:22.441 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 10u netlink 0t0 20257 KOBJECT_UEVENT 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 11u netlink 0t0 20258 GENERIC 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 12u netlink 0t0 20259 GENERIC 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 13u netlink 0t0 20260 ROUTE 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 14r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 15r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 16u netlink 0t0 20614 KOBJECT_UEVENT 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 17u a_inode 0,14 0 1057 [eventpoll] 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 18u unix 0xffff964f040aa800 0t0 18821 type=STREAM (CONNECTED) 2024-04-15T22:59:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 19u a_inode 0,14 0 1057 [eventpoll:21,27] 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 20w FIFO 0,24 0t0 1306 /run/systemd/inhibit/1.ref 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 21u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 23u raw6 0t0 18101 00000000000000000000000000000000:003A->00000000000000000000000000000000:0000 st=07 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 749 gmain root 27u IPv4 18122 0t0 UDP smithi037.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root txt REG 8,1 3644720 11633 /usr/sbin/NetworkManager 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.443 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 317552 52828 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-settings-plugin-ifcfg-rh.so 2024-04-15T22:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-04-15T22:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 62456 9782 /usr/lib64/libjansson.so.4.14.0 2024-04-15T22:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 31792 10954 /usr/lib64/libteamdctl.so.0.1.5 2024-04-15T22:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 57016 52811 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-device-plugin-team.so 2024-04-15T22:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.444 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 138712 9890 /usr/lib64/libbrotlicommon.so.1.0.9 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-04-15T22:59:22.445 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-04-15T22:59:22.446 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 53048 9892 /usr/lib64/libbrotlidec.so.1.0.9 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 77488 9888 /usr/lib64/libpsl.so.5.3.3 2024-04-15T22:59:22.447 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 483672 11481 /usr/lib64/libssh.so.4.9.4 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 169704 11377 /usr/lib64/libnghttp2.so.14.20.1 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.449 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 662304 11485 /usr/lib64/libcurl.so.4.7.0 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.450 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 32408 11334 /usr/lib64/libndp.so.0.2.0 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.451 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 1u unix 0xffff964f06a6df00 0t0 20197 type=STREAM (CONNECTED) 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 2u unix 0xffff964f06a6df00 0t0 20197 type=STREAM (CONNECTED) 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 3u a_inode 0,14 0 1057 [eventfd:6] 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 4u a_inode 0,14 0 1057 [eventfd:7] 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 5u unix 0xffff964f06a79e00 0t0 20252 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.452 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 6u unix 0xffff964f06a6ee00 0t0 20253 type=STREAM (CONNECTED) 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 7u a_inode 0,14 0 1057 [eventfd:8] 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 8r REG 0,4 0 4026531840 net 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 9r REG 0,4 0 4026532495 mnt 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 10u netlink 0t0 20257 KOBJECT_UEVENT 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 11u netlink 0t0 20258 GENERIC 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 12u netlink 0t0 20259 GENERIC 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 13u netlink 0t0 20260 ROUTE 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 14r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 15r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 16u netlink 0t0 20614 KOBJECT_UEVENT 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 17u a_inode 0,14 0 1057 [eventpoll] 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 18u unix 0xffff964f040aa800 0t0 18821 type=STREAM (CONNECTED) 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 19u a_inode 0,14 0 1057 [eventpoll:21,27] 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 20w FIFO 0,24 0t0 1306 /run/systemd/inhibit/1.ref 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 21u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 23u raw6 0t0 18101 00000000000000000000000000000000:003A->00000000000000000000000000000000:0000 st=07 2024-04-15T22:59:22.454 INFO:teuthology.orchestra.run.smithi037.stdout:NetworkMa 721 750 gdbus root 27u IPv4 18122 0t0 UDP smithi037.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2024-04-15T22:59:22.456 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.456 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.456 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root txt REG 8,1 61904 10507 /usr/sbin/irqbalance 2024-04-15T22:59:22.457 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.457 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.457 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.457 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root DEL REG 8,1 9998 /usr/lib64/libnuma.so.1.0.0 2024-04-15T22:59:22.458 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.458 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.458 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.459 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.459 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 1u unix 0xffff964f18370500 0t0 20557 type=STREAM (CONNECTED) 2024-04-15T22:59:22.459 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 2u unix 0xffff964f18370500 0t0 20557 type=STREAM (CONNECTED) 2024-04-15T22:59:22.459 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 3u a_inode 0,14 0 1057 [eventfd:1] 2024-04-15T22:59:22.460 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 4u a_inode 0,14 0 1057 [eventfd:2] 2024-04-15T22:59:22.460 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 5u unix 0xffff964f06a6b200 0t0 20221 /run/irqbalance/irqbalance724.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.460 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 root 6u unix 0xffff964f06a7ee00 0t0 23623 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.460 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.461 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.462 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root txt REG 8,1 61904 10507 /usr/sbin/irqbalance 2024-04-15T22:59:22.462 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.464 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.464 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.464 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root DEL REG 8,1 9998 /usr/lib64/libnuma.so.1.0.0 2024-04-15T22:59:22.464 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.465 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.466 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.466 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.466 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 1u unix 0xffff964f18370500 0t0 20557 type=STREAM (CONNECTED) 2024-04-15T22:59:22.466 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 2u unix 0xffff964f18370500 0t0 20557 type=STREAM (CONNECTED) 2024-04-15T22:59:22.467 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 3u a_inode 0,14 0 1057 [eventfd:1] 2024-04-15T22:59:22.467 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 4u a_inode 0,14 0 1057 [eventfd:2] 2024-04-15T22:59:22.467 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 5u unix 0xffff964f06a6b200 0t0 20221 /run/irqbalance/irqbalance724.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.467 INFO:teuthology.orchestra.run.smithi037.stdout:irqbalanc 724 731 gmain root 6u unix 0xffff964f06a7ee00 0t0 23623 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.468 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.468 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.468 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt txt REG 8,1 32280 11292 /usr/bin/lsmd 2024-04-15T22:59:22.468 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.469 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt mem REG 8,1 56912 10811 /usr/lib64/libconfig.so.11.0.2 2024-04-15T22:59:22.469 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.469 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.470 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt 1u unix 0xffff964f06a69400 0t0 20206 type=STREAM (CONNECTED) 2024-04-15T22:59:22.470 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt 2u unix 0xffff964f06a69400 0t0 20206 type=STREAM (CONNECTED) 2024-04-15T22:59:22.470 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt 3uW REG 0,24 0 1260 /run/lsm/ipc/.lsmd-ipc-lock 2024-04-15T22:59:22.470 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt 5u unix 0xffff964f06a68000 0t0 20216 /var/run/lsm/ipc/simc type=STREAM (LISTEN) 2024-04-15T22:59:22.471 INFO:teuthology.orchestra.run.smithi037.stdout:lsmd 725 libstoragemgmt 6u unix 0xffff964f18180a00 0t0 16930 /var/run/lsm/ipc/sim type=STREAM (LISTEN) 2024-04-15T22:59:22.471 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.471 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.471 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root txt REG 8,1 194568 12461 /usr/sbin/mcelog 2024-04-15T22:59:22.472 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.472 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.472 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.472 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root 1u unix 0xffff964f06a6d500 0t0 20209 type=STREAM (CONNECTED) 2024-04-15T22:59:22.473 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root 2u unix 0xffff964f06a6d500 0t0 20209 type=STREAM (CONNECTED) 2024-04-15T22:59:22.473 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root 3r CHR 10,227 0t0 142 /dev/mcelog 2024-04-15T22:59:22.473 INFO:teuthology.orchestra.run.smithi037.stdout:mcelog 726 root 4u unix 0xffff964f06a69e00 0t0 20219 /var/run/mcelog-client type=STREAM (LISTEN) 2024-04-15T22:59:22.473 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.473 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.474 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.474 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.475 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.475 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.475 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.475 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.475 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.476 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.476 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.476 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.476 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.477 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.477 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.477 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.477 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.477 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.478 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.478 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.478 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.478 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.479 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.479 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.479 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.479 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.479 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.480 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.480 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.480 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.480 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.480 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.481 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.481 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.481 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.481 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.482 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.482 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.482 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.482 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.482 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.483 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.483 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.483 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.483 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.484 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.484 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.484 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.485 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.486 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.487 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 739 gmain polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.489 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.490 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.491 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.492 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 740 gdbus polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.493 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.494 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.495 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.496 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.497 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 741 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.498 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.499 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.500 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.501 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 742 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.503 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.504 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.505 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.506 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.507 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 743 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.508 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.509 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.511 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 744 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.512 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.515 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.516 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 745 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.517 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.518 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.519 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.520 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.521 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 746 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.522 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.523 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.524 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.525 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.526 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 747 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.527 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.528 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.528 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.528 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.528 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.528 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.528 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.529 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.530 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.531 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 748 JS\x20Hel polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.534 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.535 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 4u a_inode 0,14 0 1057 [eventfd:3] 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 5u a_inode 0,14 0 1057 [eventfd:4] 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 6u unix 0xffff964f18181400 0t0 16941 type=STREAM (CONNECTED) 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 7u a_inode 0,14 0 1057 [eventfd:5] 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.536 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 9u a_inode 0,14 0 1057 [eventfd:10] 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:polkitd 727 753 polkitd polkitd 10u unix 0xffff964f1817a800 0t0 16942 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root txt REG 8,1 306896 20362 /usr/lib/systemd/systemd-logind 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 597770 1027 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.538 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.538 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.538 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.538 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.538 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.540 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 1u unix 0xffff964f18377300 0t0 20567 type=STREAM (CONNECTED) 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 2u unix 0xffff964f18377300 0t0 20567 type=STREAM (CONNECTED) 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 3u unix 0xffff964f18379e00 0t0 20592 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.541 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 4u a_inode 0,14 0 1057 [eventpoll:5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21] 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 5u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 6u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 7r REG 0,21 4096 22260 /sys/devices/virtual/tty/tty0/active 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 8u netlink 0t0 20603 KOBJECT_UEVENT 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 9r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 10r FIFO 0,24 0t0 1306 /run/systemd/inhibit/1.ref 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 11u netlink 0t0 20604 KOBJECT_UEVENT 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 12u netlink 0t0 20605 KOBJECT_UEVENT 2024-04-15T22:59:22.542 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 13u netlink 0t0 20606 KOBJECT_UEVENT 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 14u unix 0xffff964f18375a00 0t0 20607 type=STREAM (CONNECTED) 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 15u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 16u CHR 13,65 0t0 138 /dev/input/event1 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 17u CHR 13,64 0t0 137 /dev/input/event0 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 18u CHR 13,66 0t0 148 /dev/input/event2 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 19u CHR 4,6 0t0 25 /dev/tty6 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 20r FIFO 0,24 0t0 1800 /run/systemd/sessions/4.ref 2024-04-15T22:59:22.543 INFO:teuthology.orchestra.run.smithi037.stdout:systemd-l 730 root 21r FIFO 0,24 0t0 4551 /run/systemd/sessions/17.ref 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.544 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.545 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.545 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.546 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.547 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.547 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.548 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.549 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.549 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.549 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.549 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.550 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root 1u unix 0xffff964f06a6bc00 0t0 20410 type=STREAM (CONNECTED) 2024-04-15T22:59:22.550 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root 2u unix 0xffff964f06a6bc00 0t0 20410 type=STREAM (CONNECTED) 2024-04-15T22:59:22.550 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root 3u IPv4 22107 0t0 TCP *:ssh (LISTEN) 2024-04-15T22:59:22.551 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 783 root 4u IPv6 22109 0t0 TCP *:ssh (LISTEN) 2024-04-15T22:59:22.551 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.551 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.551 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root txt REG 8,1 15448 10498 /usr/bin/python3.9 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 130464 38212 /usr/lib64/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 107736 38215 /usr/lib64/python3.9/lib-dynload/_datetime.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 204696 37013 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.552 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 143200 10195 /usr/lib64/libgirepository-1.0.so.1.0.0 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 357112 39324 /usr/lib64/python3.9/site-packages/gi/_gi.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.553 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.554 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.554 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 24736 39204 /usr/lib64/python3.9/site-packages/_dbus_glib_bindings.so 2024-04-15T22:59:22.554 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.554 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.554 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.555 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.555 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.555 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.555 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.556 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-04-15T22:59:22.556 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 171376 39203 /usr/lib64/python3.9/site-packages/_dbus_bindings.so 2024-04-15T22:59:22.556 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.556 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.556 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 71408 38249 /usr/lib64/python3.9/lib-dynload/pyexpat.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.556 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.557 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 23800 38230 /usr/lib64/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.557 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 46560 38224 /usr/lib64/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.557 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 76792 9701 /usr/lib64/libbz2.so.1.0.8 2024-04-15T22:59:22.557 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 29224 41980 /usr/lib64/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.557 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 41752 38253 /usr/lib64/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.557 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 50232 41979 /usr/lib64/python3.9/lib-dynload/_blake2.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.558 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.558 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 20000 42001 /usr/lib64/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.558 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 69048 38220 /usr/lib64/python3.9/lib-dynload/_hashlib.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.558 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 24160 41992 /usr/lib64/python3.9/lib-dynload/_random.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.558 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 15984 38200 /usr/lib64/python3.9/lib-dynload/_bisect.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 124184 41989 /usr/lib64/python3.9/lib-dynload/_pickle.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 23832 38245 /usr/lib64/python3.9/lib-dynload/fcntl.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 15560 38227 /usr/lib64/python3.9/lib-dynload/_opcode.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 56424 38237 /usr/lib64/python3.9/lib-dynload/_struct.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 68000 38241 /usr/lib64/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.559 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 37592 38251 /usr/lib64/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.560 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 28056 41985 /usr/lib64/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.560 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 3486240 13688 /usr/lib64/libpython3.9.so.1.0 2024-04-15T22:59:22.560 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 67528 42002 /usr/lib64/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.560 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 101208 41993 /usr/lib64/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.560 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 20528 41991 /usr/lib64/python3.9/lib-dynload/_queue.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 1u unix 0xffff964f040ab700 0t0 18867 type=STREAM (CONNECTED) 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 2u unix 0xffff964f040ab700 0t0 18867 type=STREAM (CONNECTED) 2024-04-15T22:59:22.561 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 3w REG 8,1 10371 1216 /var/log/tuned/tuned.log 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 4u sock 0,8 0t0 13949 protocol: NETLINK 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 5u unix 0xffff964f18731e00 0t0 22126 type=STREAM (CONNECTED) 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 6u a_inode 0,14 0 1057 [eventfd:17] 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 7u unix 0xffff964f0308e900 0t0 22578 type=STREAM (CONNECTED) 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 root 8u unix 0xffff964f0308f300 0t0 22579 type=STREAM (CONNECTED) 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.562 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.563 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root txt REG 8,1 15448 10498 /usr/bin/python3.9 2024-04-15T22:59:22.563 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-04-15T22:59:22.563 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 130464 38212 /usr/lib64/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.563 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 107736 38215 /usr/lib64/python3.9/lib-dynload/_datetime.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.563 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 204696 37013 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2024-04-15T22:59:22.564 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.564 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.564 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.564 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.565 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-04-15T22:59:22.565 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-04-15T22:59:22.565 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.565 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-04-15T22:59:22.565 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 143200 10195 /usr/lib64/libgirepository-1.0.so.1.0.0 2024-04-15T22:59:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 357112 39324 /usr/lib64/python3.9/site-packages/gi/_gi.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 24736 39204 /usr/lib64/python3.9/site-packages/_dbus_glib_bindings.so 2024-04-15T22:59:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-04-15T22:59:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 171376 39203 /usr/lib64/python3.9/site-packages/_dbus_bindings.so 2024-04-15T22:59:22.567 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 71408 38249 /usr/lib64/python3.9/lib-dynload/pyexpat.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 23800 38230 /usr/lib64/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 46560 38224 /usr/lib64/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 76792 9701 /usr/lib64/libbz2.so.1.0.8 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 29224 41980 /usr/lib64/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 41752 38253 /usr/lib64/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.568 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 50232 41979 /usr/lib64/python3.9/lib-dynload/_blake2.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 20000 42001 /usr/lib64/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 69048 38220 /usr/lib64/python3.9/lib-dynload/_hashlib.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 24160 41992 /usr/lib64/python3.9/lib-dynload/_random.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 15984 38200 /usr/lib64/python3.9/lib-dynload/_bisect.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 124184 41989 /usr/lib64/python3.9/lib-dynload/_pickle.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 23832 38245 /usr/lib64/python3.9/lib-dynload/fcntl.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.569 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 15560 38227 /usr/lib64/python3.9/lib-dynload/_opcode.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 56424 38237 /usr/lib64/python3.9/lib-dynload/_struct.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 68000 38241 /usr/lib64/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 37592 38251 /usr/lib64/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 28056 41985 /usr/lib64/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 3486240 13688 /usr/lib64/libpython3.9.so.1.0 2024-04-15T22:59:22.570 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 67528 42002 /usr/lib64/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.571 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 101208 41993 /usr/lib64/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.571 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.571 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.571 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.571 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 20528 41991 /usr/lib64/python3.9/lib-dynload/_queue.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 1u unix 0xffff964f040ab700 0t0 18867 type=STREAM (CONNECTED) 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 2u unix 0xffff964f040ab700 0t0 18867 type=STREAM (CONNECTED) 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 3w REG 8,1 10371 1216 /var/log/tuned/tuned.log 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 4u sock 0,8 0t0 13949 protocol: NETLINK 2024-04-15T22:59:22.572 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 5u unix 0xffff964f18731e00 0t0 22126 type=STREAM (CONNECTED) 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 6u a_inode 0,14 0 1057 [eventfd:17] 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 7u unix 0xffff964f0308e900 0t0 22578 type=STREAM (CONNECTED) 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:tuned 784 869 tuned root 8u unix 0xffff964f0308f300 0t0 22579 type=STREAM (CONNECTED) 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-04-15T22:59:22.573 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.574 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-04-15T22:59:22.575 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.576 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.576 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.576 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.576 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.576 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-04-15T22:59:22.576 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-04-15T22:59:22.577 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.578 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 1u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 2u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 3u unix 0xffff964f1837ee00 0t0 20691 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 4r FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 5u a_inode 0,14 0 1057 [eventpoll:4,6,7,8] 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 6u a_inode 0,14 0 1057 [eventfd:16] 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 7u unix 0xffff964f18377800 0t0 20692 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 8u unix 0xffff964f18375f00 0t0 20693 /run/gssproxy.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.579 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 root 9w FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.580 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.582 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.582 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-04-15T22:59:22.582 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-04-15T22:59:22.582 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-04-15T22:59:22.582 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.582 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-04-15T22:59:22.583 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.584 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 1u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 2u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 3u unix 0xffff964f1837ee00 0t0 20691 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 4r FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 5u a_inode 0,14 0 1057 [eventpoll:4,6,7,8] 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 6u a_inode 0,14 0 1057 [eventfd:16] 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 7u unix 0xffff964f18377800 0t0 20692 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 8u unix 0xffff964f18375f00 0t0 20693 /run/gssproxy.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 789 gssproxy root 9w FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-04-15T22:59:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.587 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.587 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.587 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.587 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-04-15T22:59:22.587 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.588 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.589 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.590 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-04-15T22:59:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.591 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 1u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 2u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 3u unix 0xffff964f1837ee00 0t0 20691 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.593 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 4r FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.593 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 5u a_inode 0,14 0 1057 [eventpoll:4,6,7,8] 2024-04-15T22:59:22.593 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 6u a_inode 0,14 0 1057 [eventfd:16] 2024-04-15T22:59:22.593 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 7u unix 0xffff964f18377800 0t0 20692 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.593 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 8u unix 0xffff964f18375f00 0t0 20693 /run/gssproxy.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 790 gssproxy root 9w FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.594 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.595 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.596 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.596 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-04-15T22:59:22.596 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-04-15T22:59:22.596 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-04-15T22:59:22.596 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.596 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-04-15T22:59:22.597 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-04-15T22:59:22.598 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.598 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.598 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-04-15T22:59:22.598 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.599 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 1u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 2u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 3u unix 0xffff964f1837ee00 0t0 20691 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 4r FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 5u a_inode 0,14 0 1057 [eventpoll:4,6,7,8] 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 6u a_inode 0,14 0 1057 [eventfd:16] 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 7u unix 0xffff964f18377800 0t0 20692 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 8u unix 0xffff964f18375f00 0t0 20693 /run/gssproxy.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.600 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 791 gssproxy root 9w FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.602 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.603 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.603 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-04-15T22:59:22.603 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-04-15T22:59:22.603 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-04-15T22:59:22.603 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.603 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-04-15T22:59:22.604 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.605 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.606 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.606 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.606 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 1u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 2u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 3u unix 0xffff964f1837ee00 0t0 20691 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 4r FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 5u a_inode 0,14 0 1057 [eventpoll:4,6,7,8] 2024-04-15T22:59:22.607 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 6u a_inode 0,14 0 1057 [eventfd:16] 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 7u unix 0xffff964f18377800 0t0 20692 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 8u unix 0xffff964f18375f00 0t0 20693 /run/gssproxy.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 792 gssproxy root 9w FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.608 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.609 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.609 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.609 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-04-15T22:59:22.609 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-04-15T22:59:22.609 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.610 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.610 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.610 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.610 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-04-15T22:59:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.613 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.613 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-04-15T22:59:22.613 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.613 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.614 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.614 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.614 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.614 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.614 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.615 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.615 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.615 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 1u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.615 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 2u unix 0xffff964f18326400 0t0 18066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.617 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 3u unix 0xffff964f1837ee00 0t0 20691 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.617 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 4r FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.617 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 5u a_inode 0,14 0 1057 [eventpoll:4,6,7,8] 2024-04-15T22:59:22.617 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 6u a_inode 0,14 0 1057 [eventfd:16] 2024-04-15T22:59:22.617 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 7u unix 0xffff964f18377800 0t0 20692 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 8u unix 0xffff964f18375f00 0t0 20693 /run/gssproxy.sock type=STREAM (LISTEN) 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:gssproxy 788 793 gssproxy root 9w FIFO 0,13 0t0 20696 pipe 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root cwd DIR 8,1 4096 1657 /var/spool/at 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root txt REG 8,1 32680 12473 /usr/sbin/atd 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.618 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.619 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:atd 795 root 3uW REG 0,24 4 1329 /run/atd.pid 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root txt REG 8,1 74552 11234 /usr/sbin/crond 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.620 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.621 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.622 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root 1u unix 0xffff964f18321900 0t0 18088 type=STREAM (CONNECTED) 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root 2u unix 0xffff964f18321900 0t0 18088 type=STREAM (CONNECTED) 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root 3uW REG 0,24 4 1328 /run/crond.pid 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root 4u unix 0xffff964f18343200 0t0 18091 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:crond 796 root 5r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.623 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe txt REG 8,1 80984 12959 /usr/sbin/nrpe 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.624 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe 3u IPv4 18158 0t0 TCP *:nrpe (LISTEN) 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe 4u unix 0xffff964f18344100 0t0 18150 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:nrpe 890 nrpe 5u IPv6 18159 0t0 TCP *:nrpe (LISTEN) 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root txt REG 8,1 58072 13690 /usr/sbin/agetty 2024-04-15T22:59:22.625 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root 0u CHR 4,1 0t0 20 /dev/tty1 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root 1u CHR 4,1 0t0 20 /dev/tty1 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root 2u CHR 4,1 0t0 20 /dev/tty1 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1319 root 4r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.626 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 1320 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 1320 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 1320 root txt unknown /proc/1320/exe 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root txt REG 8,1 58072 13690 /usr/sbin/agetty 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.628 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root 0u CHR 4,65 0t0 114 /dev/ttyS1 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root 1u CHR 4,65 0t0 114 /dev/ttyS1 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root 2u CHR 4,65 0t0 114 /dev/ttyS1 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:agetty 1326 root 4r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.629 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu DEL REG 8,1 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu DEL REG 8,1 1425 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.630 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.631 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.632 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.633 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 1u unix 0xffff964f18731900 0t0 22248 type=STREAM (CONNECTED) 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 2u unix 0xffff964f18731900 0t0 22248 type=STREAM (CONNECTED) 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 3u unix 0xffff964f18738a00 0t0 22276 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 4u a_inode 0,14 0 1057 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 5u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 6r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 7r DIR 0,25 0 3393 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 8u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.634 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 9u a_inode 0,14 0 1057 [eventpoll:10,13] 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 10r REG 0,20 0 22265 /proc/2039/mountinfo 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 11r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 12u unix 0xffff964f18322800 0t0 18329 /run/user/1000/bus type=STREAM (LISTEN) 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 13r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 14r REG 0,20 0 4026532076 /proc/swaps 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 15u netlink 0t0 18311 KOBJECT_UEVENT 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 16u unix 0xffff964f18346400 0t0 18316 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.635 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 17u unix 0xffff964f18346900 0t0 18317 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 18u unix 0xffff964f18340500 0t0 18318 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 19u unix 0xffff964f18327800 0t0 18319 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 20u unix 0xffff964f18324b00 0t0 18321 type=STREAM (CONNECTED) 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:systemd 2039 ubuntu 21u a_inode 0,14 0 1057 [timerfd] 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-04-15T22:59:22.636 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu DEL REG 8,1 63926 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-04-15T22:59:22.637 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.638 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.640 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu DEL REG 8,1 63928 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.641 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 1u unix 0xffff964f18731900 0t0 22248 type=STREAM (CONNECTED) 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 2u unix 0xffff964f18731900 0t0 22248 type=STREAM (CONNECTED) 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 3u a_inode 0,14 0 1057 [eventfd:13] 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 4u a_inode 0,14 0 1057 [eventfd:14] 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 6w FIFO 0,13 0t0 22252 pipe 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:(sd-pam) 2048 ubuntu 7u unix 0xffff964f1873a800 0t0 22268 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.642 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root txt REG 8,1 769928 12237 /usr/sbin/rsyslogd 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 33592 12207 /usr/lib64/rsyslog/imjournal.so 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 42248 12214 /usr/lib64/rsyslog/imuxsock.so 2024-04-15T22:59:22.643 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 32808 12215 /usr/lib64/rsyslog/lmnet.so 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 36328 9712 /usr/lib64/libuuid.so.1.3.0 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.644 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 52856 12201 /usr/lib64/libfastjson.so.4.3.0 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 16032 12200 /usr/lib64/libestr.so.0.0.0 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 3r CHR 1,9 0t0 9 /dev/urandom 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 4u unix 0xffff964f18395000 0t0 26798 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 5r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 7r REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 8w REG 8,1 653687 1642 /var/log/secure 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 9r REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 root 10w REG 8,1 4934187 1641 /var/log/messages 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root txt REG 8,1 769928 12237 /usr/sbin/rsyslogd 2024-04-15T22:59:22.646 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 33592 12207 /usr/lib64/rsyslog/imjournal.so 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 42248 12214 /usr/lib64/rsyslog/imuxsock.so 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 32808 12215 /usr/lib64/rsyslog/lmnet.so 2024-04-15T22:59:22.647 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 36328 9712 /usr/lib64/libuuid.so.1.3.0 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 52856 12201 /usr/lib64/libfastjson.so.4.3.0 2024-04-15T22:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 16032 12200 /usr/lib64/libestr.so.0.0.0 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 3r CHR 1,9 0t0 9 /dev/urandom 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 4u unix 0xffff964f18395000 0t0 26798 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.649 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 5r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 7r REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 8w REG 8,1 653687 1642 /var/log/secure 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 9r REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5110 in:imjour root 10w REG 8,1 4934187 1641 /var/log/messages 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root txt REG 8,1 769928 12237 /usr/sbin/rsyslogd 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.650 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 33592 12207 /usr/lib64/rsyslog/imjournal.so 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 42248 12214 /usr/lib64/rsyslog/imuxsock.so 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 32808 12215 /usr/lib64/rsyslog/lmnet.so 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.651 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 36328 9712 /usr/lib64/libuuid.so.1.3.0 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 52856 12201 /usr/lib64/libfastjson.so.4.3.0 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 16032 12200 /usr/lib64/libestr.so.0.0.0 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 3r CHR 1,9 0t0 9 /dev/urandom 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 4u unix 0xffff964f18395000 0t0 26798 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 5r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 7r REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.653 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 8w REG 8,1 653687 1642 /var/log/secure 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 9r REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:rsyslogd 5108 5111 rs:main root 10w REG 8,1 4934187 1641 /var/log/messages 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:tls-strp 15844 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:tls-strp 15844 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:tls-strp 15844 root txt unknown /proc/15844/exe 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.654 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root txt REG 8,1 630408 11631 /usr/sbin/smartd 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.655 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.656 INFO:teuthology.orchestra.run.smithi037.stdout:smartd 15892 root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21249 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21249 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21249 root txt unknown /proc/21249/exe 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21278 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21278 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21278 root txt unknown /proc/21278/exe 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21307 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.657 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21307 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21307 root txt unknown /proc/21307/exe 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21336 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21336 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21336 root txt unknown /proc/21336/exe 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21365 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21365 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:kdmflush/ 21365 root txt unknown /proc/21365/exe 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:xfsalloc 21530 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.658 INFO:teuthology.orchestra.run.smithi037.stdout:xfsalloc 21530 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfsalloc 21530 root txt unknown /proc/21530/exe 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs_mru_c 21531 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs_mru_c 21531 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs_mru_c 21531 root txt unknown /proc/21531/exe 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-buf/d 21532 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-buf/d 21532 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-buf/d 21532 root txt unknown /proc/21532/exe 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-conv/ 21533 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.659 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-conv/ 21533 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-conv/ 21533 root txt unknown /proc/21533/exe 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-recla 21534 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-recla 21534 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-recla 21534 root txt unknown /proc/21534/exe 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-block 21535 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-block 21535 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-block 21535 root txt unknown /proc/21535/exe 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-inode 21536 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.660 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-inode 21536 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-inode 21536 root txt unknown /proc/21536/exe 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-log/d 21537 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-log/d 21537 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-log/d 21537 root txt unknown /proc/21537/exe 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-cil/d 21538 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-cil/d 21538 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfs-cil/d 21538 root txt unknown /proc/21538/exe 2024-04-15T22:59:22.661 INFO:teuthology.orchestra.run.smithi037.stdout:xfsaild/d 21539 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:xfsaild/d 21539 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:xfsaild/d 21539 root txt unknown /proc/21539/exe 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 23101 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 23101 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/4 23101 root txt unknown /proc/23101/exe 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.662 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 19520 32283 /usr/lib64/security/pam_lastlog.so 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-04-15T22:59:22.663 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15264 32290 /usr/lib64/security/pam_motd.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-04-15T22:59:22.664 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 19368 32301 /usr/lib64/security/pam_sepermit.so 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15232 32292 /usr/lib64/security/pam_nologin.so 2024-04-15T22:59:22.665 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.666 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.667 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.668 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 3u unix 0xffff964f18345a00 0t0 86655 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 4u IPv4 89628 0t0 TCP smithi037.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:45774 (ESTABLISHED) 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 6u unix 0xffff964f18320a00 0t0 86630 type=STREAM (CONNECTED) 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 7w FIFO 0,24 0t0 1800 /run/systemd/sessions/4.ref 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23467 root 8u unix 0xffff964f18321e00 0t0 86659 type=STREAM (CONNECTED) 2024-04-15T22:59:22.669 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 19520 32283 /usr/lib64/security/pam_lastlog.so 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-04-15T22:59:22.670 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15264 32290 /usr/lib64/security/pam_motd.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-04-15T22:59:22.671 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 19368 32301 /usr/lib64/security/pam_sepermit.so 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.672 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15232 32292 /usr/lib64/security/pam_nologin.so 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.673 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.674 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-04-15T22:59:22.675 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 1u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 3u unix 0xffff964f18345a00 0t0 86655 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 4u IPv4 89628 0t0 TCP smithi037.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:45774 (ESTABLISHED) 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 5u unix 0xffff964f18320f00 0t0 86658 type=STREAM (CONNECTED) 2024-04-15T22:59:22.676 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 6u unix 0xffff964f18320a00 0t0 86630 type=STREAM (CONNECTED) 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 7w FIFO 0,24 0t0 1800 /run/systemd/sessions/4.ref 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 12r FIFO 0,13 0t0 110053 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 13r FIFO 0,13 0t0 157672 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 14r FIFO 0,13 0t0 110054 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 15r FIFO 0,13 0t0 176197 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 16r FIFO 0,13 0t0 157673 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 17r FIFO 0,13 0t0 180892 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 18r FIFO 0,13 0t0 176198 pipe 2024-04-15T22:59:22.677 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 19r FIFO 0,13 0t0 232856 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 20r FIFO 0,13 0t0 180893 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 21r FIFO 0,13 0t0 229869 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 22r FIFO 0,13 0t0 231167 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 24r FIFO 0,13 0t0 229870 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 25r FIFO 0,13 0t0 232857 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:sshd 23471 ubuntu 26r FIFO 0,13 0t0 231168 pipe 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.678 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.679 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony txt REG 8,1 353408 12490 /usr/sbin/chronyd 2024-04-15T22:59:22.679 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-04-15T22:59:22.679 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.679 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.680 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 2u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 3u unix 0xffff964f18630f00 0t0 97447 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.681 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 5u IPv4 97452 0t0 UDP localhost:323 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 6u IPv6 97453 0t0 UDP localhost:323 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:chronyd 23531 chrony 7u unix 0xffff964f18631900 0t0 97454 /run/chrony/chronyd.sock type=DGRAM (CONNECTED) 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph txt REG 8,1 15448 10498 /usr/bin/python3.9 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 46560 38224 /usr/lib64/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 76792 9701 /usr/lib64/libbz2.so.1.0.8 2024-04-15T22:59:22.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 41752 38253 /usr/lib64/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 68000 38241 /usr/lib64/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 101208 41993 /usr/lib64/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 3486240 13688 /usr/lib64/libpython3.9.so.1.0 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 29224 41980 /usr/lib64/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 20000 42001 /usr/lib64/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 28056 41985 /usr/lib64/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 23800 38230 /usr/lib64/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 37592 38251 /usr/lib64/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 67528 42002 /usr/lib64/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph 1u unix 0xffff964f2229c100 0t0 96066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-cras 23746 ceph 2u unix 0xffff964f2229c100 0t0 96066 type=STREAM (CONNECTED) 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 25158 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 25158 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/u 25158 root txt unknown /proc/25158/exe 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 25396 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 25396 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/1 25396 root txt unknown /proc/25396/exe 2024-04-15T22:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 26137 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 26137 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/2 26137 root txt unknown /proc/26137/exe 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 26733 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 26733 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/3 26733 root txt unknown /proc/26733/exe 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 27795 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 27795 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.686 INFO:teuthology.orchestra.run.smithi037.stdout:kworker/6 27795 root txt unknown /proc/27795/exe 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root txt REG 8,1 185296 11759 /usr/bin/sudo 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.687 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.688 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-04-15T22:59:22.689 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-04-15T22:59:22.690 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 472568 54513 /usr/libexec/sudo/sudoers.so 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 119832 54509 /usr/libexec/sudo/libsudo_util.so.0.0.0 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.693 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 0r FIFO 0,13 0t0 110052 pipe 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 1w FIFO 0,13 0t0 110053 pipe 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 2w FIFO 0,13 0t0 110054 pipe 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 3r FIFO 0,13 0t0 105922 pipe 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 4w FIFO 0,13 0t0 105922 pipe 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 5u netlink 0t0 105927 AUDIT 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:sudo 28184 root 6u unix 0xffff964f03074b00 0t0 105928 type=DGRAM (CONNECTED) 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.694 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root txt REG 8,1 90216 11090 /usr/bin/journalctl 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.695 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-04-15T22:59:22.698 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.699 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root 0r FIFO 0,13 0t0 110052 pipe 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root 1w FIFO 0,13 0t0 110053 pipe 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root 2w FIFO 0,13 0t0 110054 pipe 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root 3r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root 5r REG 0,24 16777216 2073 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:journalct 28201 root 6r REG 0,24 6395200 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@f9006c4d1fed421ab7df56ab406c027d-0000000000000001-0006162a520c44d2.journal 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:22.700 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:22.702 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:22.703 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 3u unix 0xffff964f18341e00 0t0 112995 type=SEQPACKET (CONNECTED) 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 4r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 5u a_inode 0,14 0 1057 [eventfd:15] 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 6w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.704 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 7r FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 9r FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 10u unix 0xffff964f18635500 0t0 206573 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 11r FIFO 8,1 0t0 530324 /var/lib/containers/storage/overlay-containers/f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c/userdata/ctl 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 12u unix 0xffff964f1817cb00 0t0 111181 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 13w FIFO 8,1 0t0 530324 /var/lib/containers/storage/overlay-containers/f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c/userdata/ctl 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 14r FIFO 8,1 0t0 530325 /var/lib/containers/storage/overlay-containers/f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c/userdata/winsz 2024-04-15T22:59:22.705 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 15w FIFO 8,1 0t0 530325 /var/lib/containers/storage/overlay-containers/f39d1d96ec93a6abb0057a6be52855d188c510c5afc52c43210a2f0e489f4a7c/userdata/winsz 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29187 root 16u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root txt REG 8,1 810088 1960 /run/podman-init 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29189 root 3u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:22.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29192 log ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29193 service ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29194 admin_soc ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29195 signal_ha ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29196 rocksdb:l ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29197 rocksdb:h ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29213 ceph-mon ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29214 fn_monsto ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29215 msgr-work ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29216 msgr-work ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29217 msgr-work ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29218 OpHistory ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29219 fin ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29220 safe_time ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29221 cpu_tp ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29222 cpu_tp ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29223 cpu_tp ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29224 cpu_tp ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29225 ms_dispat ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29226 ms_local ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29227 safe_time ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:22.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:22.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:22.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:22.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:22.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:22.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:22.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:22.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:22.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:22.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29228 ms_dispat ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:22.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:22.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:23.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:23.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:23.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 29229 ms_local ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph cwd DIR 0,42 4096 530309 / 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph rtd DIR 0,42 4096 530309 / 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph txt REG 0,42 13489776 412462 /usr/bin/ceph-mon 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 412462 /usr/bin/ceph-mon (path inode=2803) 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422279 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265659) 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422679 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422604 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406000 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406113 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph mem REG 8,1 422275 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265655) 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 0u CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 1w FIFO 0,13 0t0 111179 pipe 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 2w FIFO 0,13 0t0 111180 pipe 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 3r FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 4w FIFO 0,13 0t0 111979 pipe 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 5u IPv4 114702 0t0 TCP smithi037.front.sepia.ceph.com:3300 (LISTEN) 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 6r FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 7w FIFO 0,13 0t0 111991 pipe 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 8u unix 0xffff964f18735a00 0t0 111992 /var/run/ceph/ceph-mon.smithi037.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 9r FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 10w FIFO 0,13 0t0 111993 pipe 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 11r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 12uW REG 253,4 0 33816709 /var/lib/ceph/mon/ceph-smithi037/store.db/LOCK 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 13r DIR 253,4 4096 33816708 /var/lib/ceph/mon/ceph-smithi037/store.db 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 14u IPv4 210691 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:56522 (ESTABLISHED) 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 15w REG 253,4 444568 33842736 /var/lib/ceph/mon/ceph-smithi037/store.db/000021.log 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 17w REG 253,4 761 33816715 /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000015 2024-04-15T22:59:23.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 18u a_inode 0,14 0 1057 [eventpoll:5,19,28,40,43,45,49,51] 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 19r FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 20w FIFO 0,13 0t0 112002 pipe 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 21u a_inode 0,14 0 1057 [eventpoll:14,22,41,44,47] 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 22r FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 23w FIFO 0,13 0t0 112003 pipe 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 24u a_inode 0,14 0 1057 [eventpoll:25,38,42,46,48] 2024-04-15T22:59:23.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 25r FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 26w FIFO 0,13 0t0 112004 pipe 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 27w REG 8,1 27413981 530343 /var/log/ceph/ceph-mon.smithi037.log 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 28u IPv4 114703 0t0 TCP smithi037.front.sepia.ceph.com:smc-https (LISTEN) 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 29r FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 30w FIFO 0,13 0t0 110420 pipe 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 31r FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 32w FIFO 0,13 0t0 110421 pipe 2024-04-15T22:59:23.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 33r FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 34w FIFO 0,13 0t0 110422 pipe 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 35u unix 0xffff964f1873d500 0t0 112010 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 36w REG 8,1 58648 530344 /var/log/ceph/ceph.log 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 37w REG 8,1 141911 530349 /var/log/ceph/ceph.audit.log 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 38u IPv4 145658 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59316 (ESTABLISHED) 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 39w REG 8,1 13281 530399 /var/log/ceph/ceph.cephadm.log 2024-04-15T22:59:23.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 40u IPv4 138941 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59322 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 41u IPv4 145671 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59350 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 42u IPv4 138949 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:59360 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 43u IPv4 140066 0t0 TCP smithi037.front.sepia.ceph.com:60326->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 44u IPv4 149712 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:38394 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 45u IPv4 147175 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi189.front.sepia.ceph.com:33896 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 46u IPv4 159403 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:45552 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 47u IPv4 183019 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49326 (ESTABLISHED) 2024-04-15T22:59:23.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 48u IPv4 175400 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:39140 (ESTABLISHED) 2024-04-15T22:59:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 49u IPv4 185060 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:49342 (ESTABLISHED) 2024-04-15T22:59:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 50r REG 253,4 4949648 33842738 /var/lib/ceph/mon/ceph-smithi037/store.db/000023.sst 2024-04-15T22:59:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon 29191 59293 rstore_co ceph 51u IPv4 211362 0t0 TCP smithi037.front.sepia.ceph.com:3300->smithi037.front.sepia.ceph.com:41722 (ESTABLISHED) 2024-04-15T22:59:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root cwd DIR 8,1 4096 2 / 2024-04-15T22:59:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root rtd DIR 8,1 4096 2 / 2024-04-15T22:59:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-04-15T22:59:23.026 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-15T22:59:23.027 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-04-15T22:59:23.028 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 0r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 1w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 2w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 3u unix 0xffff964f18341400 0t0 113072 type=SEQPACKET (CONNECTED) 2024-04-15T22:59:23.029 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 4r CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 5u a_inode 0,14 0 1057 [eventfd:21] 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 6w CHR 1,3 0t0 4 /dev/null 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 7r FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 8r a_inode 0,14 0 1057 inotify 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 9r FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 10u unix 0xffff964f1817ad00 0t0 111314 type=DGRAM (UNCONNECTED) 2024-04-15T22:59:23.030 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 11r FIFO 8,1 0t0 530365 /var/lib/containers/storage/overlay-containers/9fa350a659796c3f141a5f40b7290ed5482153d6109a961102679c428f517bbe/userdata/ctl 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 12u unix 0xffff964f1817b200 0t0 111265 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 13w FIFO 8,1 0t0 530365 /var/lib/containers/storage/overlay-containers/9fa350a659796c3f141a5f40b7290ed5482153d6109a961102679c428f517bbe/userdata/ctl 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 14r FIFO 8,1 0t0 530366 /var/lib/containers/storage/overlay-containers/9fa350a659796c3f141a5f40b7290ed5482153d6109a961102679c428f517bbe/userdata/winsz 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 15w FIFO 8,1 0t0 530366 /var/lib/containers/storage/overlay-containers/9fa350a659796c3f141a5f40b7290ed5482153d6109a961102679c428f517bbe/userdata/winsz 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:conmon 29422 root 16u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.031 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root txt REG 8,1 810088 1960 /run/podman-init 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:podman-in 29424 root 3u a_inode 0,14 0 1057 [signalfd] 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 5w REG 8,1 8520765 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37206 log ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.099 INFO:tasks.workunit.client.0.smithi037.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-15T22:59:23.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37208 msgr-work ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37209 msgr-work ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37210 msgr-work ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37214 service ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37215 admin_soc ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37216 ceph_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37217 fn_anonym ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37218 signal_ha ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37219 mgrsb-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37220 ms_dispat ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37221 ms_local ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37222 io_contex ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37223 io_contex ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37224 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37225 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37226 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37227 fn_anonym ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37228 flusher ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37229 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.670 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.672 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.674 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37294 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37295 ms_dispat ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37296 ms_local ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37297 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37298 cmdfin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.820 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.821 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37302 io_contex ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.822 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.823 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.824 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.825 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.826 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.827 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.828 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.829 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.830 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37303 io_contex ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37304 ceph_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37305 ms_dispat ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37306 ms_local ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37307 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.973 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:23.974 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:23.975 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:23.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:23.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:23.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:23.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.976 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:23.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:23.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:23.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:23.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:23.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:23.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:23.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:23.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:23.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:23.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:23.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:23.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:23.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37308 safe_time ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:23.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:23.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:23.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:23.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:23.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37309 balancer ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37310 m-fin-bal ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37311 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37312 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.118 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.119 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.120 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.121 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.122 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.123 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.124 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.125 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.127 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.128 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.129 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.130 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37313 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.132 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.133 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.134 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.135 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.136 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.138 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.139 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.140 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.142 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.143 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.144 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.146 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.147 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.148 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.149 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.150 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.151 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.152 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.153 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.154 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.155 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.156 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.157 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37314 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.159 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.161 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.162 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.163 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.164 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.165 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.166 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.167 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.168 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.169 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.170 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.171 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.172 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.173 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.174 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.175 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.176 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.177 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.178 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.179 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.180 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.182 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.184 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.186 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37315 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.188 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.189 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.190 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.191 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.192 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.195 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.196 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.197 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.198 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.199 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.200 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.201 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.203 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.204 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.205 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.206 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.207 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.208 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.209 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.210 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.211 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.212 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.213 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.214 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37316 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.215 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.216 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.217 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.218 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.219 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.220 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.221 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.222 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.223 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.224 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.225 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.226 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.227 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.228 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.229 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.230 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.232 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.233 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.234 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.235 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.236 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.237 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.238 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.239 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.240 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.241 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37317 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.243 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.244 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.245 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.246 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.247 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.248 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.249 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.250 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.251 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.252 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.253 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.254 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.255 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.257 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.258 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.259 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.262 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.264 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.265 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.266 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.267 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.268 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37318 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.269 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.270 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.271 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.272 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.273 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.274 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.275 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.276 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.277 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.278 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.279 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.280 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.281 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.282 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.283 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.284 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.285 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.286 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.287 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.288 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.289 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.290 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.291 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.292 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37319 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.296 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.297 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.298 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.299 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.300 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.301 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.303 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.305 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.306 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.307 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.308 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.309 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.310 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.311 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.313 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.314 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.315 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.316 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37320 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.318 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.319 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.320 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.321 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.322 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.323 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.326 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.328 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.329 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.330 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.331 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.332 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.334 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.335 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.336 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.337 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.338 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.339 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.340 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37321 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.341 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.342 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.343 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.344 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.345 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.346 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.347 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.348 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.349 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.350 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.351 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.352 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.353 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.354 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.356 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.357 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.359 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.360 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.361 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.362 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.363 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.364 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37322 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.366 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.367 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.368 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.369 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.370 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.371 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.372 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.374 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.375 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.376 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.377 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.378 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.379 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.380 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.381 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.382 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.383 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.384 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.385 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.386 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.387 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.388 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37323 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.389 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.390 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.391 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.392 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.393 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.394 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.395 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.396 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.397 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.398 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.399 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.400 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.401 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.402 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.403 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.404 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.405 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.406 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.407 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.408 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.409 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.410 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.411 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.412 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37324 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.413 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.414 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.415 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.416 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.418 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.419 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.420 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.421 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.422 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.423 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.424 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.425 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.426 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.427 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.428 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.429 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.430 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.431 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.434 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.436 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37325 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.441 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.443 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.444 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.445 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.446 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.447 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.448 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.449 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.450 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.451 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.452 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.454 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.455 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.456 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.457 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.458 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.460 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37326 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.461 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.464 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.465 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.466 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.468 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.469 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.470 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.471 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.472 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.473 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.474 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.476 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.477 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.478 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.479 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.480 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.481 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.482 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37327 cephadm ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.485 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.486 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.489 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.490 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.491 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.492 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.493 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.494 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.495 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.497 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.498 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.499 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.500 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.501 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.502 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.503 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.504 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.505 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.507 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37328 m-fin-cep ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.508 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.511 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.512 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.514 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.515 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.516 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.517 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.518 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.519 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.520 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.521 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.522 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.523 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.524 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.525 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.526 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.527 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.528 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.529 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.530 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.531 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37329 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.532 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.533 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.534 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.535 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.536 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.537 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.539 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.540 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.542 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.543 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.544 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.545 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.546 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.548 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.550 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.551 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.553 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.554 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.555 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37330 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.556 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.557 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.558 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.559 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.561 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.563 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.564 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.565 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.566 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.567 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.569 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.570 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.571 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.572 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.573 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.574 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.576 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.577 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.579 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37331 cephadm ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.580 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.583 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.585 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.586 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.587 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.588 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.589 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.590 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.591 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.592 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.593 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.594 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.595 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.596 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.597 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.598 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.599 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.600 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.601 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.602 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37332 crash ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.605 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.607 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.608 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.609 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.610 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.611 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.612 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.613 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.614 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.615 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.618 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.619 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.620 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.621 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.622 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.624 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.625 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.626 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.627 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37333 m-fin-cra ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.628 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.629 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.631 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.632 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.635 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.636 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.637 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.638 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.639 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.640 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.641 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.642 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.643 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.646 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.647 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.648 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.649 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.650 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.651 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37334 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.652 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.653 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.656 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.657 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.658 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.659 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.660 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.662 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.663 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.664 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.665 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.666 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.667 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.668 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.669 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.671 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 22:59:23 smithi189 ceph-mon[36179]: pgmap v247: 65 pgs: 65 active+clean; 146 MiB data, 717 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 834 op/s 2024-04-15T22:59:24.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.675 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.676 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.677 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.679 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.680 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37335 dashboard ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.681 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.682 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.683 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.684 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.685 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.686 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.687 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.688 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.689 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.690 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.691 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.692 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.693 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.694 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.696 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.697 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.698 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.699 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.701 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.702 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.703 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.704 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37336 m-fin-das ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.706 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.707 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.708 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.709 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.711 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.712 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.713 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.714 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.715 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.716 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.717 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.718 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.719 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.720 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.721 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.722 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.723 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.724 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.725 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.726 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.728 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37337 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.729 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.730 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.731 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.732 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.734 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.735 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.736 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.737 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.738 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.739 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.740 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.741 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.742 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.743 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.744 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.745 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.747 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.750 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.751 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.752 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37338 devicehea ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.753 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.754 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.755 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.756 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.757 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.760 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.762 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.763 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.764 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.765 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.766 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.767 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.768 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.769 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.770 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.771 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.773 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.774 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.776 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37339 m-fin-dev ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.777 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.778 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.779 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.780 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.781 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.782 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.783 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.784 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.785 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.786 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.787 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.788 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.789 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.790 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.791 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.792 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.793 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.796 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.797 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.798 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.799 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.800 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37340 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.801 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.802 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.803 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.804 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.805 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.806 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.807 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.810 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.811 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.812 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.813 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.815 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.817 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.818 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:alertmanager.smithi037 smithi037 *:9093,9094 running (6m) 106s ago 7m 25.2M - 0.25.0 c8568f914cd2 e049eff81a04 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter.smithi037 smithi037 running (8m) 106s ago 8m 7424k - 18.2.0 dc2bc1663786 3616d122dfb5 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-exporter.smithi189 smithi189 running (6m) 111s ago 6m 7348k - 18.2.0 dc2bc1663786 4b1aba4b2eba 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:crash.smithi037 smithi037 running (8m) 106s ago 8m 6665k - 18.2.0 dc2bc1663786 944063fe7d85 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:crash.smithi189 smithi189 running (6m) 111s ago 6m 6660k - 18.2.0 dc2bc1663786 f483fc8d01e0 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:grafana.smithi037 smithi037 *:3000 running (6m) 106s ago 7m 83.6M - 9.4.7 954c08fa6188 a7a1c1b0e042 2024-04-15T22:59:24.819 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi037.ifpdxj smithi037 running (2m) 106s ago 2m 18.0M - 18.2.0 dc2bc1663786 e360d43528ac 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi037.rdehay smithi037 running (2m) 106s ago 2m 11.3M - 18.2.0 dc2bc1663786 f6a0a7e18f69 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi189.kzokod smithi189 running (2m) 111s ago 2m 12.0M - 18.2.0 dc2bc1663786 9a555801f8f6 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mds.cephfs.smithi189.pqjbmb smithi189 running (2m) 111s ago 2m 15.6M - 18.2.0 dc2bc1663786 616697835d13 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.smithi037.gtwkck smithi037 *:9283,8765,8443 running (9m) 106s ago 9m 487M - 18.2.0 dc2bc1663786 9fa350a65979 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mgr.smithi189.lanqnp smithi189 *:8443,9283,8765 running (6m) 111s ago 6m 438M - 18.2.0 dc2bc1663786 a9d94b5f083f 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mon.smithi037 smithi037 running (9m) 106s ago 9m 49.1M 2048M 18.2.0 dc2bc1663786 f39d1d96ec93 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:mon.smithi189 smithi189 running (6m) 111s ago 6m 36.1M 2048M 18.2.0 dc2bc1663786 e2331fdccbaa 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter.smithi037 smithi037 *:9100 running (8m) 106s ago 8m 16.0M - 1.5.0 0da6a335fe13 a4d1b4d06e69 2024-04-15T22:59:24.820 INFO:teuthology.orchestra.run.smithi037.stdout:node-exporter.smithi189 smithi189 *:9100 running (6m) 111s ago 6m 15.0M - 1.5.0 0da6a335fe13 f9d0ca18f1f6 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 smithi037 running (5m) 106s ago 5m 48.3M 3948M 18.2.0 dc2bc1663786 45282f17f939 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 smithi037 running (5m) 106s ago 5m 50.7M 3948M 18.2.0 dc2bc1663786 7c4e631bf214 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 smithi037 running (4m) 106s ago 4m 48.6M 3948M 18.2.0 dc2bc1663786 2c1c43054c1a 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 smithi189 running (4m) 111s ago 4m 46.0M 4970M 18.2.0 dc2bc1663786 5e54e0afe09f 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 smithi189 running (4m) 111s ago 4m 44.3M 4970M 18.2.0 dc2bc1663786 c6102fb44c70 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 smithi189 running (3m) 111s ago 3m 46.9M 4970M 18.2.0 dc2bc1663786 e4f59c36ac6f 2024-04-15T22:59:24.821 INFO:teuthology.orchestra.run.smithi037.stdout:prometheus.smithi037 smithi037 *:9095 running (5m) 106s ago 7m 62.5M - 2.43.0 a07b618ecd1d a2075b4529f9 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.904+0000 7f7349e2b700 1 -- 172.21.15.37:0/3755126579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7344100350 msgr2=0x7f73441007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.904+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3755126579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7344100350 0x7f73441007a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7338009910 tx=0x7f7338009c20 comp rx=0 tx=0).stop 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.904+0000 7f7349e2b700 1 -- 172.21.15.37:0/3755126579 shutdown_connections 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.904+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3755126579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7344100350 0x7f73441007a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.904+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3755126579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73440ff150 0x7f73440ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.904+0000 7f7349e2b700 1 -- 172.21.15.37:0/3755126579 >> 172.21.15.37:0/3755126579 conn(0x7f73440fa700 msgr2=0x7f73440fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.905+0000 7f7349e2b700 1 -- 172.21.15.37:0/3755126579 shutdown_connections 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.905+0000 7f7349e2b700 1 -- 172.21.15.37:0/3755126579 wait complete. 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.905+0000 7f7349e2b700 1 Processor -- start 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f7349e2b700 1 -- start start 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f7349e2b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 0x7f73441949a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f7349e2b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7344100350 0x7f7344194ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f7349e2b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7344195500 con 0x7f73440ff150 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f7349e2b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7344195640 con 0x7f7344100350 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f73437fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 0x7f73441949a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f73437fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 0x7f73441949a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51904/0 (socket says 172.21.15.37:51904) 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f73437fe700 1 -- 172.21.15.37:0/3261475210 learned_addr learned my addr 172.21.15.37:0/3261475210 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.906+0000 7f7342ffd700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7344100350 0x7f7344194ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f73437fe700 1 -- 172.21.15.37:0/3261475210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7344100350 msgr2=0x7f7344194ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f73437fe700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7344100350 0x7f7344194ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f73437fe700 1 -- 172.21.15.37:0/3261475210 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73380095f0 con 0x7f73440ff150 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f7342ffd700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7344100350 0x7f7344194ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f73437fe700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 0x7f73441949a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f73340098f0 tx=0x7f7334009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7334012070 con 0x7f73440ff150 2024-04-15T22:59:24.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f73340044d0 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7344068e30 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7334019480 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.907+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7344069320 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.909+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7330005320 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.909+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f7334016030 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.910+0000 7f7340ff9700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f732c06c680 0x7f732c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.910+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f733408cb50 con 0x7f73440ff150 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.910+0000 7f7342ffd700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f732c06c680 0x7f732c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T22:59:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.911+0000 7f7342ffd700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f732c06c680 0x7f732c06eb30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7338006010 tx=0x7f733800b540 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T22:59:24.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:22.917+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f733405ace0 con 0x7f73440ff150 2024-04-15T22:59:24.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.184+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 --> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7330000bf0 con 0x7f732c06c680 2024-04-15T22:59:24.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.195+0000 7f7340ff9700 1 -- 172.21.15.37:0/3261475210 <== mgr.14223 v2:172.21.15.37:6800/935132805 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3456 (secure 0 0 0) 0x7f7330000bf0 con 0x7f732c06c680 2024-04-15T22:59:24.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.197+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f732c06c680 msgr2=0x7f732c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:24.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.197+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f732c06c680 0x7f732c06eb30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7338006010 tx=0x7f733800b540 comp rx=0 tx=0).stop 2024-04-15T22:59:24.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 msgr2=0x7f73441949a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 0x7f73441949a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f73340098f0 tx=0x7f7334009c00 comp rx=0 tx=0).stop 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 shutdown_connections 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:6800/935132805,v1:172.21.15.37:6801/935132805] conn(0x7f732c06c680 0x7f732c06eb30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73440ff150 0x7f73441949a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 --2- 172.21.15.37:0/3261475210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7344100350 0x7f7344194ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 >> 172.21.15.37:0/3261475210 conn(0x7f73440fa700 msgr2=0x7f7344103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 shutdown_connections 2024-04-15T22:59:24.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-15T22:59:23.198+0000 7f7349e2b700 1 -- 172.21.15.37:0/3261475210 wait complete. 2024-04-15T22:59:24.832 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.833 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.834 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.835 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.836 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.837 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.838 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37341 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.841 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.842 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.843 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.844 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.845 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.846 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.847 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.848 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.849 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.850 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.851 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.852 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.853 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.854 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.855 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.856 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.857 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.858 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.859 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.860 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.861 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.862 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37343 m-fin-ios ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.864 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.865 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.868 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.869 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.870 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.873 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.874 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.875 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.876 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.877 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.878 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.879 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.880 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.881 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.883 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.884 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.885 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.886 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37345 m-fin-nfs ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.887 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.888 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.889 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.890 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.891 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.892 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.893 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.894 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.895 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.896 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.897 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.899 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.900 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.901 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.902 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.903 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.905 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.906 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.907 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.908 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.909 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.910 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37346 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.911 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.912 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.915 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.917 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.918 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.920 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.921 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.922 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.923 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.924 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.925 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.926 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.927 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.929 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.930 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.931 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.932 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.933 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37347 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.935 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.936 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.937 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.938 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.939 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.940 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.942 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.943 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.944 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.945 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.946 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.947 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.948 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.949 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.952 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.953 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.954 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.955 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.956 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.957 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37348 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.958 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.959 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.960 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.961 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.963 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.964 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.965 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.966 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.967 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.968 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.969 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.970 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.971 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:24.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.972 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:24.972 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Apr 15 22:59:23 smithi037 ceph-mon[29191]: pgmap v247: 65 pgs: 65 active+clean; 146 MiB data, 717 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 834 op/s 2024-04-15T22:59:24.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:24.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:24.977 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:24.978 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:24.979 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:24.981 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:24.982 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:24.983 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.984 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:24.985 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37350 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:24.987 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.988 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.989 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:24.990 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:24.991 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.992 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.993 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.994 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:24.995 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:24.998 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:24.999 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:25.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.002 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:25.003 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.004 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:25.005 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:25.006 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:25.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.008 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:25.009 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:25.010 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37351 m-fin-orc ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:25.011 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:25.012 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.014 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.015 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:25.016 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.018 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.019 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.020 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:25.022 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.024 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.025 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:25.026 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.027 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:25.029 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.030 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:25.031 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.032 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:25.033 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.034 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37352 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:25.035 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.036 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:25.037 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.039 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.040 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.041 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.042 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.043 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.044 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:25.047 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.048 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.049 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:25.050 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.053 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.054 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:25.055 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.056 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.058 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:25.059 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37353 pg_autosc ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:25.060 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.061 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.062 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.063 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.064 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.065 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.067 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.069 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:25.070 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:25.071 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:25.073 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:25.074 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:25.075 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.076 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:25.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:25.079 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:25.081 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.082 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:25.083 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37354 mgr-fin ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.084 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:25.085 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.086 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.087 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.088 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.090 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.091 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.092 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:25.093 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405897 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405975 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405987 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405959 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406120 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.094 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406033 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422791 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422799 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406019 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406160 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422924 /usr/lib64/libunwind.so.8.0.1 (path inode=2324) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406157 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406097 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422843 /usr/lib64/librados.so.2.0.0 (path inode=1771) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405890 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.095 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405934 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406005 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406122 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422914 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405908 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406168 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422855 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422714 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-04-15T22:59:25.096 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406150 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406078 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405904 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405885 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406087 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405895 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422286 /usr/lib64/ceph/libceph-common.so.2 (path inode=265472) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405914 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422896 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-04-15T22:59:25.097 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406082 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 406118 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 405861 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422549 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph mem REG 8,1 422628 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 0u CHR 1,3 0t0 5 /dev/null 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 1w FIFO 0,13 0t0 111263 pipe 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 2w FIFO 0,13 0t0 111264 pipe 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 3r FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.098 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 4w FIFO 0,13 0t0 138907 pipe 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 5w REG 8,1 8518893 530383 /var/log/ceph/ceph-mgr.smithi037.gtwkck.log 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 6u a_inode 0,14 0 1057 [eventpoll:7,27,28,37,52,53,57,62,64,66,67,73,78,80] 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 7r FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 8w FIFO 0,13 0t0 138908 pipe 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 9u a_inode 0,14 0 1057 [eventpoll:10,22,29,56,58,59,63,65,70,74,76,79] 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 10r FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 11w FIFO 0,13 0t0 138909 pipe 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 12u a_inode 0,14 0 1057 [eventpoll:13,40,54,55,60,61,69,71,72,75,77,81] 2024-04-15T22:59:25.099 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 13r FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 14w FIFO 0,13 0t0 138910 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 15r FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 16w FIFO 0,13 0t0 140005 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 17u unix 0xffff964f6a0c8500 0t0 140006 /var/run/ceph/ceph-mgr.smithi037.gtwkck.asok type=STREAM (LISTEN) 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 18r FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 19w FIFO 0,13 0t0 140010 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 20r FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 21w FIFO 0,13 0t0 140011 pipe 2024-04-15T22:59:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 22u IPv4 141092 0t0 TCP smithi037.front.sepia.ceph.com:59316->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 23r FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 24w FIFO 0,13 0t0 141909 pipe 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 25r FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 26w FIFO 0,13 0t0 141910 pipe 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 27u IPv4 141093 0t0 TCP *:6800 (LISTEN) 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 28u IPv4 141095 0t0 TCP *:acnet (LISTEN) 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 29u IPv4 144656 0t0 TCP smithi037.front.sepia.ceph.com:59322->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 30u REG 0,46 98 556088 /tmp/cephadm-conf-e2xebpuw 2024-04-15T22:59:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 31u REG 0,46 2635 556089 /tmp/cephadm-identity-jyc1wxev 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 32w REG 0,46 595 556090 /tmp/cephadm-identity-jyc1wxev.pub 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 33u a_inode 0,14 0 1057 [eventpoll:34,39,42] 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 34u unix 0xffff964f2229e400 0t0 145664 type=STREAM (CONNECTED) 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 35u unix 0xffff964f22299e00 0t0 145665 type=STREAM (CONNECTED) 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 36u REG 0,46 1034 556091 /tmp/tmpr4it8vl2 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 37u IPv4 141098 0t0 TCP smithi037.front.sepia.ceph.com:59350->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 38u REG 0,46 1704 556092 /tmp/tmpbkphecbt 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 39u IPv4 138952 0t0 TCP smithi037.front.sepia.ceph.com:36022->smithi037.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 40u IPv4 141914 0t0 TCP smithi037.front.sepia.ceph.com:59360->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 41u IPv6 221772 0t0 TCP *:callwaveiam (LISTEN) 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 42u IPv4 138953 0t0 TCP smithi037.front.sepia.ceph.com:58968->smithi189.front.sepia.ceph.com:ssh (ESTABLISHED) 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 43u a_inode 0,14 0 1057 [eventpoll:41,68] 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 44u IPv6 145675 0t0 TCP *:pcsync-https (LISTEN) 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 45u a_inode 0,14 0 1057 [eventpoll:44] 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 46u REG 0,46 1761 556093 /tmp/tmpgbzwjvxe 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 47u REG 0,46 3243 556094 /tmp/tmp9340vsv3 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 48u IPv4 146786 0t0 TCP smithi037.front.sepia.ceph.com:7150 (LISTEN) 2024-04-15T22:59:25.103 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 49u a_inode 0,14 0 1057 [eventpoll:48] 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 50u IPv4 145742 0t0 TCP smithi037.front.sepia.ceph.com:ultraseek-http (LISTEN) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 51u a_inode 0,14 0 1057 [eventpoll:50] 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 52u IPv4 141151 0t0 TCP smithi037.front.sepia.ceph.com:60290->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 53u IPv4 141152 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60290 (ESTABLISHED) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 54u IPv4 142842 0t0 TCP smithi037.front.sepia.ceph.com:60296->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 55u IPv4 141155 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60296 (ESTABLISHED) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 56u IPv4 144682 0t0 TCP smithi037.front.sepia.ceph.com:60302->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 57u IPv4 141156 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60302 (ESTABLISHED) 2024-04-15T22:59:25.104 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 58u IPv4 144683 0t0 TCP smithi037.front.sepia.ceph.com:60310->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 59u IPv4 141159 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60310 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 60u IPv4 141162 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:60326 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 61u IPv4 161457 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:34702 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 62u IPv4 151592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38718 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 63u IPv4 151640 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38736 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 64u IPv4 172500 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:45322 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 65u IPv4 184145 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49006 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 66u IPv4 191592 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53680 (ESTABLISHED) 2024-04-15T22:59:25.105 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 67u IPv4 184246 0t0 TCP smithi037.front.sepia.ceph.com:49014->smithi037.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 68u IPv6 223675 0t0 TCP smithi037.front.sepia.ceph.com:callwaveiam->smithi037.front.sepia.ceph.com:35120 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 69u IPv4 180974 0t0 TCP smithi037.front.sepia.ceph.com:49342->smithi037.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 70u IPv4 184247 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49014 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 71u IPv4 186219 0t0 TCP smithi037.front.sepia.ceph.com:52404->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 72u IPv4 180975 0t0 TCP smithi037.front.sepia.ceph.com:52420->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 73u IPv4 184250 0t0 TCP smithi037.front.sepia.ceph.com:35402->smithi037.front.sepia.ceph.com:6802 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 74u IPv4 184287 0t0 TCP smithi037.front.sepia.ceph.com:52428->smithi037.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-04-15T22:59:25.106 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 75u IPv4 181042 0t0 TCP smithi037.front.sepia.ceph.com:42134->smithi037.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 76u IPv4 193681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:46020 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 77u IPv4 211681 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:49610 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 78u IPv4 212452 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:53268 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 79u IPv4 206835 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:40616 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 80u IPv4 187065 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi189.front.sepia.ceph.com:38822 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37355 m-fin-pg_ ceph 81u IPv4 212360 0t0 TCP smithi037.front.sepia.ceph.com:6800->smithi037.front.sepia.ceph.com:57750 (ESTABLISHED) 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph cwd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph rtd DIR 0,46 4096 530350 / 2024-04-15T22:59:25.107 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph txt REG 0,46 5484952 412461 /usr/bin/ceph-mgr 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 412461 /usr/bin/ceph-mgr (path inode=2781) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406042 /usr/lib64/libnss_dns-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406044 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422802 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407969 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406031 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424403 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424265 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.108 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424003 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424135 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424137 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423697 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422839 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422688 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422809 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422811 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423603 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.109 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422946 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422741 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422948 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 426274 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423384 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405955 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405937 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406164 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422673 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-04-15T22:59:25.110 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422951 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423390 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407932 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406105 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405973 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405913 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405906 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422849 /usr/lib64/librbd.so.1.18.0 (path inode=1775) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424352 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.111 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405925 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407963 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407925 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406166 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422991 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423150 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407914 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407941 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407952 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.112 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407970 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407943 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423491 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407931 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407936 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407949 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407954 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407956 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422627 /usr/lib64/libcephfs.so.2.0.0 (path inode=2295) 2024-04-15T22:59:25.113 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422986 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423000 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 423013 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407945 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407915 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407946 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407916 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407934 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407957 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.114 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406002 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407939 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405889 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407917 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407972 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407950 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405929 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407926 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.115 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407948 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407947 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407929 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 424346 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407958 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407966 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407944 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407937 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407942 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.116 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 407935 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422930 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422928 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422926 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 422759 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2260) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406067 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405981 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 405989 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-04-15T22:59:25.117 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr 29426 37356 progress ceph mem REG 8,1 406103 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-04-15T22:59:45.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:44.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:45.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:44.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:45.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:44.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:46.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:45.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:46.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:45.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:46.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:45.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:46.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:46.115+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:46.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:46.115+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:46.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:46.115+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:47.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:46.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:47.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:46.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:47.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:46.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:47.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:46.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:47.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:46.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:47.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:46.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:47.437 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:47.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:47.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:47.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:47.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:47.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:48.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:48.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:48.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:48.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:48.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:48.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:48.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:48.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:48.212 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:48.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:48.212 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:48.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:48.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:47.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:48.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:47.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:48.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:47.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:49.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:49.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:49.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:49.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:49.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:49.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:49.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:49.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:49.212 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:49.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:49.212 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:49.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:49.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:48.896+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:49.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:48.896+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:49.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:48.896+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:50.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:49.902+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:50.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:49.902+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:50.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:49.902+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:50.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:49.992+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:50.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:49.992+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:50.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:49.992+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:50.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:50.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:50.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:50.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:50.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:50.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:51.202 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:50.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:51.202 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:50.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:51.202 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:50.938+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:51.203 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:50.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:51.203 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:50.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:51.203 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:50.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:51.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:51.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:51.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:51.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:51.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:52.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:52.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:52.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:52.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:52.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:52.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:52.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:51.931+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:52.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:51.931+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:52.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:51.931+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:52.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:52.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:52.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:52.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:52.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:52.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:53.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:53.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:53.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:53.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:53.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:53.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:53.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:52.912+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:53.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:52.912+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:53.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:52.912+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:53.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:52.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:53.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:52.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:53.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:52.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:54.145+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:54.145+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:54.145+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:54.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:53.946+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:54.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:53.946+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:54.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:53.946+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:54.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:54.013+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:54.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:54.013+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:54.212 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:54.014+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:55.173 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:55.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:55.173 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:55.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:55.174 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:55.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:55.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:54.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:55.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:54.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:55.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:54.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:55.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:55.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:55.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:55.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:55.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:55.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:56.204 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:55.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:56.204 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:55.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:56.204 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:55.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:56.204 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:56.086+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:56.204 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:56.086+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:56.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:56.086+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:56.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:56.213+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:56.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:56.213+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:56.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:56.213+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:57.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:56.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:57.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:56.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:57.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:56.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:57.188 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:57.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:57.188 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:57.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:57.188 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:57.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:57.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:57.196+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:57.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:57.196+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:57.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:57.196+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:58.145 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:57.861+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:58.146 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:57.861+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:58.146 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:57.861+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:58.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:58.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:58.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:58.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:58.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:58.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:58.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:58.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:58.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:58.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:58.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:58.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:59.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:58.846+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:59.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:58.846+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:59.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:58.846+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T22:59:59.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:59.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:59.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:59.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:59.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T22:59:59.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T22:59:59.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:59.130+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:59.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:59.130+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T22:59:59.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T22:59:59.130+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:00.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:59.850+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:00.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:59.850+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:00.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 22:59:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T22:59:59.850+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:00.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:00.143+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:00.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:00.143+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:00.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:00.143+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:00.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:00.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:00.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:00.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:00.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:00.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:01.136 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:00.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:01.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:00.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:01.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:00.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:01.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:01.244+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:01.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:01.244+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:01.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:01.244+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:01.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:01.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:01.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:01.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:01.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:01.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:02.111 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:01.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:02.111 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:01.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:02.111 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:01.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:02.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:02.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:02.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:02.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:02.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:02.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:02.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:02.120+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:02.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:02.120+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:02.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:02.120+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:02.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:02.821+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:02.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:02.821+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:02.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:02.821+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:03.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:03.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:03.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:03.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:03.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:03.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:03.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:03.153+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:03.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:03.153+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:03.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:03.153+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:04.129 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:03.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:04.129 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:03.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:04.129 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:03.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:04.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:04.138+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:04.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:04.138+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:04.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:04.138+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:04.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:04.142+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:04.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:04.142+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:04.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:04.142+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:05.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:04.873+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:05.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:04.873+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:05.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:04.873+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:05.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:05.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:05.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:05.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:05.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:05.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:05.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:05.149+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:05.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:05.149+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:05.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:05.149+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:06.121 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:05.836+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:06.121 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:05.836+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:06.122 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:05.836+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:06.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:06.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:06.463 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:06.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:06.463 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:06.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:06.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:06.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:06.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:06.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:06.464 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:06.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:07.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:06.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:07.164 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:06.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:07.164 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:06.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:07.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:07.173+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:07.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:07.173+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:07.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:07.173+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:07.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:07.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:07.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:07.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:07.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:07.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:07.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:07.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:07.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:07.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:07.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:07.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:08.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:08.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:08.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:08.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:08.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:08.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:08.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:08.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:08.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:08.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:08.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:08.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:09.144 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:08.824+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:09.145 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:08.825+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:09.145 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:08.825+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:09.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:09.154+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:09.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:09.154+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:09.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:09.154+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:09.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:09.248+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:09.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:09.248+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:09.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:09.248+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:10.118 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:09.851+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:10.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:09.851+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:10.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:09.851+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:10.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:10.266+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:10.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:10.266+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:10.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:10.266+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:10.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:10.128+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:10.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:10.128+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:10.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:10.128+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:11.135 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:10.852+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:11.136 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:10.852+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:11.136 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:10.852+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:11.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:11.145+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:11.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:11.145+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:11.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:11.145+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:11.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:11.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:11.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:11.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:11.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:11.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:12.155 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:11.861+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:12.156 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:11.861+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:12.156 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:11.861+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:12.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:12.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:12.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:12.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:12.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:12.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:12.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:12.224+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:12.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:12.224+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:12.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:12.224+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:12.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:12.845+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:12.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:12.845+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:12.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:12.845+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:13.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:13.204+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:13.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:13.204+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:13.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:13.204+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:13.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:13.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:13.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:13.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:13.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:13.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:14.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:13.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:14.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:13.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:14.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:13.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:14.619 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:14.241+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:14.619 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:14.241+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:14.619 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:14.241+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:14.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:14.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:14.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:14.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:14.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:14.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:15.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:14.920+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:15.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:14.920+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:15.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:14.920+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:15.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:15.292+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:15.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:15.292+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:15.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:15.292+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:15.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:15.285+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:15.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:15.285+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:15.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:15.285+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:16.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:15.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:16.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:15.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:16.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:15.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:16.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:16.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:16.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:16.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:16.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:16.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:16.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:16.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:16.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:16.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:16.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:16.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:17.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:16.854+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:17.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:16.854+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:17.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:16.854+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:17.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:17.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:17.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:17.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:17.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:17.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:17.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:17.255+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:17.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:17.255+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:17.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:17.255+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:18.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:17.879+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:18.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:17.879+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:18.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:17.879+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:18.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:18.321+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:18.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:18.321+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:18.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:18.321+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:18.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:18.221+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:18.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:18.221+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:18.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:18.221+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:19.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:18.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:19.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:18.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:19.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:18.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:19.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:19.348+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:19.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:19.348+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:19.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:19.348+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:19.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:19.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:19.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:19.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:19.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:19.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:20.165 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:19.922+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:20.165 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:19.922+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:20.165 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:19.922+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:20.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:20.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:20.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:20.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:20.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:20.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:20.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:20.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:20.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:20.175+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:20.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:20.175+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:21.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:20.946+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:21.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:20.946+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:21.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:20.946+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:21.212 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:21.188+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:21.212 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:21.188+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:21.213 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:21.188+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:21.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:21.360+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:21.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:21.360+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:21.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:21.360+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:22.312 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:21.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:22.313 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:21.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:22.313 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:21.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:22.313 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:22.214+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:22.313 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:22.214+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:22.313 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:22.214+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:22.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:22.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:22.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:22.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:22.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:22.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:23.193 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:23.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:23.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:23.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:23.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:23.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:23.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:23.356+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:23.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:23.356+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:23.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:23.356+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:23.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:23.202+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:23.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:23.202+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:23.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:23.202+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:24.329 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:24.183+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:24.329 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:24.183+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:24.329 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:24.183+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:24.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:24.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:24.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:24.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:24.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:24.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:24.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:24.338+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:24.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:24.338+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:24.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:24.338+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:25.355 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:25.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:25.356 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:25.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:25.356 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:25.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:25.356 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:24.981+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:25.356 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:24.981+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:25.356 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:24.981+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:25.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:25.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:25.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:25.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:25.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:25.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:26.329 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:26.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:26.329 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:26.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:26.330 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:26.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:26.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:26.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:26.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:26.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:26.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:26.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:26.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:26.338+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:26.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:26.338+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:26.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:26.338+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:27.330 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:27.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:27.330 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:27.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:27.330 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:27.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:27.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:27.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:27.330 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:27.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:27.331 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:27.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:27.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:27.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:27.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:27.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:27.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:27.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:28.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:28.083+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:28.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:28.083+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:28.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:28.083+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:28.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:28.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:28.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:28.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:28.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:28.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:28.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:28.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:28.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:28.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:28.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:28.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:29.285 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:29.109+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:29.285 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:29.109+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:29.286 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:29.109+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:29.286 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:29.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:29.286 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:29.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:29.286 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:29.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:29.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:29.294+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:29.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:29.294+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:29.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:29.294+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:30.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:30.251+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:30.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:30.251+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:30.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:30.251+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:30.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:30.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:30.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:30.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:30.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:30.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:30.463 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:30.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:30.463 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:30.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:30.463 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:30.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:31.363 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:31.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:31.363 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:31.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:31.363 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:31.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:31.363 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:31.247+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:31.364 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:31.247+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:31.364 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:31.247+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:31.364 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:31.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:31.364 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:31.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:31.364 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:31.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:32.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:32.091+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:32.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:32.091+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:32.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:32.091+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:32.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:32.291+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:32.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:32.291+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:32.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:32.291+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:32.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:32.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:32.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:32.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:32.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:32.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:33.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:33.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:33.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:33.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:33.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:33.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:33.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:33.076+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:33.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:33.076+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:33.212 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:33.076+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:33.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:33.292+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:33.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:33.292+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:33.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:33.292+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:34.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:34.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:34.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:34.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:34.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:34.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:34.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:34.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:34.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:34.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:34.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:34.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:34.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:34.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:34.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:34.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:34.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:34.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:35.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:35.154+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:35.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:35.154+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:35.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:35.154+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:35.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:35.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:35.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:35.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:35.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:35.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:35.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:35.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:35.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:35.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:35.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:35.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:36.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:36.109+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:36.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:36.109+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:36.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:36.109+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:36.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:36.258+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:36.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:36.258+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:36.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:36.258+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:36.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:36.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:36.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:36.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:36.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:36.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:37.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:37.064+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:37.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:37.064+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:37.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:37.064+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:37.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:37.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:37.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:37.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:37.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:37.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:37.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:37.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:37.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:37.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:37.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:37.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:38.192 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:38.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:38.192 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:38.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:38.192 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:38.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:38.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:38.224+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:38.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:38.224+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:38.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:38.224+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:38.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:38.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:38.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:38.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:38.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:38.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:39.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:39.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:39.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:39.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:39.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:39.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:39.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:39.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:39.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:39.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:39.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:39.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:39.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:39.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:39.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:39.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:39.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:39.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:40.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:40.014+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:40.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:40.014+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:40.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:40.014+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:40.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:40.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:40.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:40.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:40.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:40.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:40.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:40.175+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:40.463 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:40.175+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:40.463 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:40.175+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:41.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:41.054+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:41.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:41.054+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:41.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:41.054+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:41.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:41.175+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:41.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:41.175+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:41.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:41.175+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:41.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:41.150+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:41.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:41.150+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:41.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:41.150+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:42.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:42.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:42.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:42.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:42.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:42.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:42.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:42.207+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:42.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:42.207+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:42.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:42.207+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:42.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:42.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:42.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:42.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:42.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:42.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:43.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:43.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:43.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:43.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:43.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:43.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:43.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:43.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:43.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:43.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:43.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:43.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:43.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:43.067+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:43.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:43.067+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:43.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:43.067+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:44.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:44.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:44.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:44.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:44.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:44.156+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:44.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:44.156+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:44.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:44.156+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:44.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:44.029+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:44.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:44.029+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:44.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:44.029+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:45.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:45.057+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:45.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:45.057+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:45.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:45.057+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:45.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:45.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:45.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:45.178+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:45.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:45.178+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:45.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:45.037+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:45.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:45.037+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:45.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:45.037+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:46.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:46.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:46.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:46.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:46.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:46.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:46.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:46.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:46.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:46.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:46.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:46.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:46.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:46.076+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:46.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:46.076+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:46.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:46.076+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:47.028+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:47.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:47.028+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:47.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:47.028+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:47.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:47.103+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:47.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:47.103+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:47.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:47.103+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:47.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:47.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:47.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:47.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:47.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:47.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:48.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:48.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:48.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:48.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:48.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:48.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:48.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:48.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:48.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:48.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:48.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:48.198+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:48.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:48.150+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:48.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:48.150+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:48.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:48.150+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:49.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:49.104+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:49.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:49.104+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:49.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:49.104+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:49.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:49.211+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:49.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:49.211+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:49.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:49.211+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:49.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:49.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:49.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:49.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:49.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:49.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:50.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:50.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:50.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:50.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:50.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:50.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:50.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:50.141+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:50.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:50.141+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:50.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:50.141+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:50.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:50.230+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:50.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:50.230+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:50.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:50.230+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:51.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:51.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:51.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:51.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:51.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:51.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:51.275+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:51.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:51.275+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:51.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:51.275+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:51.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:51.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:51.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:51.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:51.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:51.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:52.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:52.081+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:52.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:52.081+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:52.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:52.081+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:52.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:52.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:52.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:52.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:52.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:52.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:52.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:52.262+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:52.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:52.262+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:52.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:52.262+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:53.099+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:53.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:53.099+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:53.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:53.099+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:53.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:53.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:53.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:53.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:53.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:53.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:53.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:53.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:53.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:53.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:53.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:53.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:54.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:54.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:54.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:54.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:54.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:54.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:54.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:54.212+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:54.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:54.212+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:54.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:54.212+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:54.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:54.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:54.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:54.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:54.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:54.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:55.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:55.258+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:55.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:55.258+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:55.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:55.258+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:55.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:55.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:55.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:55.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:55.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:55.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:55.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:55.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:55.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:55.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:55.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:55.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:56.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:56.211+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:56.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:56.211+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:56.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:56.211+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:56.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:56.237+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:56.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:56.237+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:56.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:56.237+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:56.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:56.090+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:56.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:56.090+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:56.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:56.090+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:57.362 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:57.185+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:57.362 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:57.185+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:57.362 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:57.185+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:57.363 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:57.268+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:57.363 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:57.268+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:57.363 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:57.268+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:57.363 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:57.101+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:57.363 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:57.101+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:57.363 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:57.101+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:58.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:58.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:58.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:58.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:58.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:58.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:58.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:58.231+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:58.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:58.231+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:58.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:58.231+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:58.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:58.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:58.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:58.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:58.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:58.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:59.108+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:59.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:59.108+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:59.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:00:59.108+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:00:59.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:59.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:59.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:59.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:59.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:00:59.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:00:59.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:59.234+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:59.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:59.234+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:00:59.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:00:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:00:59.234+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:00.100+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:00.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:00.100+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:00.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:00.100+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:00.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:00.247+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:00.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:00.247+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:00.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:00.247+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:00.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:00.205+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:00.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:00.205+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:00.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:00.205+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:01.060+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:01.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:01.060+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:01.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:01.060+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:01.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:01.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:01.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:01.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:01.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:01.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:01.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:01.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:01.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:01.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:01.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:01.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:02.306 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:02.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:02.306 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:02.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:02.307 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:02.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:02.307 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:02.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:02.307 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:02.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:02.307 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:02.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:02.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:02.316+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:02.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:02.316+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:02.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:02.316+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:03.333 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:03.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:03.334 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:03.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:03.334 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:03.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:03.334 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:03.280+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:03.334 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:03.280+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:03.334 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:03.280+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:03.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:03.343+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:03.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:03.343+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:03.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:03.343+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:04.343 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:04.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:04.344 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:04.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:04.344 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:04.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:04.344 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:04.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:04.344 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:04.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:04.344 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:04.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:04.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:04.354+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:04.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:04.354+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:04.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:04.354+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:05.375 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:05.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:05.376 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:05.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:05.376 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:05.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:05.376 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:05.197+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:05.376 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:05.197+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:05.376 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:05.197+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:05.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:05.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:05.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:05.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:05.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:05.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:06.404 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:06.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:06.404 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:06.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:06.404 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:06.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:06.404 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:06.057+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:06.405 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:06.057+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:06.405 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:06.057+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:06.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:06.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:06.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:06.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:06.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:06.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:07.383 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:07.176+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:07.383 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:07.176+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:07.383 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:07.176+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:07.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:07.028+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:07.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:07.028+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:07.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:07.028+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:07.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:07.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:07.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:07.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:07.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:07.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:08.362 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:08.074+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:08.362 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:08.074+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:08.362 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:08.074+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:08.362 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:08.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:08.363 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:08.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:08.363 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:08.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:08.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:08.372+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:08.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:08.372+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:08.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:08.372+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:09.405 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:09.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:09.405 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:09.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:09.405 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:09.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:09.405 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:09.111+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:09.405 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:09.111+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:09.406 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:09.111+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:09.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:09.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:09.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:09.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:09.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:09.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:10.369 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:10.185+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:10.369 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:10.185+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:10.369 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:10.185+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:10.370 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:10.121+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:10.370 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:10.121+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:10.370 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:10.121+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:10.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:10.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:10.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:10.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:10.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:10.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:11.380 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:11.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:11.381 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:11.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:11.381 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:11.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:11.381 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:11.196+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:11.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:11.196+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:11.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:11.196+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:11.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:11.390+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:11.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:11.390+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:11.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:11.390+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:12.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:12.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:12.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:12.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:12.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:12.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:12.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:12.344+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:12.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:12.344+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:12.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:12.344+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:12.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:12.187+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:12.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:12.187+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:12.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:12.187+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:13.627 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:13.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:13.627 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:13.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:13.627 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:13.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:13.627 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:13.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:13.628 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:13.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:13.628 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:13.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:13.630 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:13.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:13.630 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:13.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:13.630 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:13.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:14.183+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:14.183+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:14.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:14.183+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:14.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:14.214+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:14.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:14.214+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:14.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:14.214+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:14.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:14.416+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:14.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:14.416+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:14.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:14.416+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:15.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:15.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:15.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:15.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:15.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:15.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:15.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:15.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:15.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:15.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:15.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:15.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:15.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:15.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:15.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:15.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:15.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:15.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:16.372 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:16.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:16.373 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:16.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:16.373 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:16.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:16.373 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:16.211+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:16.373 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:16.211+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:16.373 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:16.211+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:16.698 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:16.381+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:16.699 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:16.381+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:16.699 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:16.381+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:17.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:17.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:17.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:17.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:17.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:17.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:17.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:17.242+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:17.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:17.242+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:17.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:17.242+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:17.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:17.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:17.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:17.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:17.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:17.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:18.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:18.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:18.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:18.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:18.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:18.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:18.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:18.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:18.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:18.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:18.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:18.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:18.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:18.268+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:18.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:18.268+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:18.462 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:18.268+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:19.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:19.316+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:19.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:19.316+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:19.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:19.316+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:19.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:19.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:19.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:19.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:19.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:19.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:19.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:19.173+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:19.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:19.173+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:19.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:19.173+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:20.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:20.294+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:20.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:20.294+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:20.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:20.294+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:20.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:20.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:20.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:20.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:20.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:20.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:20.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:20.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:20.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:20.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:20.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:20.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:21.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:21.326+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:21.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:21.326+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:21.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:21.326+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:21.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:21.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:21.461 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:21.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:21.462 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:21.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:21.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:21.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:21.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:21.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:21.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:21.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:22.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:22.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:22.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:22.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:22.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:22.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:22.463 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:22.288+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:22.463 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:22.288+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:22.463 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:22.288+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:22.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:22.371+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:22.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:22.371+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:22.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:22.371+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:23.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:23.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:23.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:23.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:23.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:23.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:23.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:23.293+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:23.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:23.293+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:23.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:23.293+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:23.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:23.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:23.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:23.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:23.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:23.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:24.415 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:24.170+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:24.416 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:24.170+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:24.416 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:24.170+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:24.416 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:24.304+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:24.416 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:24.304+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:24.416 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:24.304+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:24.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:24.425+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:24.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:24.425+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:24.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:24.425+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:25.423 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:25.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:25.423 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:25.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:25.423 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:25.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:25.423 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:25.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:25.423 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:25.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:25.424 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:25.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:25.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:25.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:25.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:25.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:25.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:25.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:26.437 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:26.300+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:26.438 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:26.300+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:26.438 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:26.300+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:26.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:26.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:26.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:26.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:26.439 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:26.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:26.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:26.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:26.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:26.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:26.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:26.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:27.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:27.337+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:27.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:27.337+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:27.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:27.337+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:27.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:27.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:27.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:27.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:27.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:27.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:27.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:27.481+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:27.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:27.481+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:27.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:27.481+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:28.433 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:28.332+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:28.434 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:28.332+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:28.434 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:28.332+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:28.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:28.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:28.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:28.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:28.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:28.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:28.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:28.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:28.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:28.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:28.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:28.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:29.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:29.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:29.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:29.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:29.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:29.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:29.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:29.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:29.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:29.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:29.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:29.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:29.712 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:29.361+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:29.712 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:29.361+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:29.712 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:29.361+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:30.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:30.375+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:30.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:30.375+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:30.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:30.375+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:30.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:30.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:30.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:30.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:30.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:30.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:30.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:30.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:30.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:30.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:30.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:30.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:31.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:31.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:31.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:31.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:31.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:31.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:31.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:31.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:31.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:31.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:31.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:31.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:31.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:31.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:31.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:31.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:31.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:31.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:32.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:01:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:01:31.866+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:01:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:32.364+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:32.364+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:32.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:32.364+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:32.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:32.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:32.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:32.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:32.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:32.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:32.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:32.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:32.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:32.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:32.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:32.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:33.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:33.490+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:33.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:33.490+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:33.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:33.490+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:33.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:33.360+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:33.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:33.360+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:33.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:33.360+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:33.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:33.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:33.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:33.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:33.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:33.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:34.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:34.504+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:34.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:34.504+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:34.711 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:34.504+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:34.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:34.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:34.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:34.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:34.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:34.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:34.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:34.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:34.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:34.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:34.712 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:34.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:35.537 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:35.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:35.537 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:35.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:35.537 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:35.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:35.538 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:35.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:35.538 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:35.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:35.538 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:35.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:35.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:35.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:35.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:35.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:35.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:35.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:36.576 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:36.295+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:36.577 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:36.295+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:36.577 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:36.295+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:36.577 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:36.342+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:36.577 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:36.342+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:36.577 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:36.342+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:36.857 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:36.587+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:36.857 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:36.587+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:36.857 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:36.587+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:37.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:01:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:01:36.867+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:01:37.561 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:37.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:37.561 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:37.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:37.562 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:37.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:37.562 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:37.348+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:37.562 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:37.348+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:37.562 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:37.348+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:37.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:37.571+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:37.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:37.571+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:37.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:37.571+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:38.566 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:38.302+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:38.566 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:38.302+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:38.566 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:38.302+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:38.567 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:38.311+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:38.567 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:38.311+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:38.567 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:38.311+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:38.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:38.576+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:38.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:38.576+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:38.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:38.576+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:39.611 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:39.345+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:39.611 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:39.345+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:39.611 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:39.345+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:39.611 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:39.304+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:39.611 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:39.304+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:39.612 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:39.304+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:39.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:39.621+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:39.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:39.621+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:39.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:39.621+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:40.658 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:40.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:40.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:40.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:40.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:40.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:40.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:40.329+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:40.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:40.329+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:40.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:40.329+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:40.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:40.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:40.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:41.662 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:41.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:41.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:41.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:41.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:41.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:41.663 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:41.353+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:41.663 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:41.353+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:41.664 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:41.353+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:41.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:01:41.867+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:01:41.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:41.672+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:41.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:41.672+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:41.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:41.672+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:42.633 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:42.271+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:42.634 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:42.271+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:42.634 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:42.271+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:42.634 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:42.398+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:42.634 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:42.398+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:42.634 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:42.398+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:42.643+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:42.643+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:42.643+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:43.569 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:43.411+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:43.569 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:43.411+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:43.569 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:43.411+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:43.570 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:43.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:43.570 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:43.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:43.570 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:43.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:43.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:43.595+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:43.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:43.595+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:43.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:43.595+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:44.605 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:44.317+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:44.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:44.317+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:44.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:44.317+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:44.606 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:44.446+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:44.606 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:44.446+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:44.606 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:44.446+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:44.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:44.616+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:44.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:44.616+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:44.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:44.616+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:45.609 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:45.363+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:45.609 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:45.363+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:45.610 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:45.363+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:45.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:45.478+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:45.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:45.478+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:45.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:45.478+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:45.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:45.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:45.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:45.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:45.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:45.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:46.582 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:46.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:46.582 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:46.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:46.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:46.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:46.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:46.338+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:46.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:46.338+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:46.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:46.338+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:46.858 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:46.592+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:46.858 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:46.592+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:46.858 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:46.592+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:47.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:01:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:01:46.868+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 18 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:01:47.615 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:47.403+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:47.616 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:47.403+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:47.616 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:47.403+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:47.616 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:47.355+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:47.616 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:47.355+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:47.616 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:47.355+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:47.626+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:47.626+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:47.626+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:48.618 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:48.310+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:48.618 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:48.310+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:48.618 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:48.310+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:48.618 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:48.451+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:48.618 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:48.451+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:48.619 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:48.451+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:48.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:48.657+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:48.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:48.657+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:48.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:48.657+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:49.609 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:49.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:49.609 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:49.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:49.609 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:49.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:49.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:49.407+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:49.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:49.407+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:49.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:49.407+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:49.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:49.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:49.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:49.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:49.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:49.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:50.589 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:50.323+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:50.589 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:50.323+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:50.590 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:50.323+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:50.590 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:50.440+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:50.591 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:50.440+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:50.591 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:50.440+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:50.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:50.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:50.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:50.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:50.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:50.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:51.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:51.289+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:51.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:51.289+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:51.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:51.289+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:51.547 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:51.490+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:51.547 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:51.490+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:51.547 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:51.490+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:51.858 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:51.556+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:51.858 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:51.556+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:51.858 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:51.556+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:52.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:01:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:01:51.868+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 36 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:01:52.554 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:52.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:52.554 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:52.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:52.554 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:52.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:52.557 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:52.293+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:52.557 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:52.293+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:52.558 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:52.293+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:52.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:52.564+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:52.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:52.564+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:52.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:52.564+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:53.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:53.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:53.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:53.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:53.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:53.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:53.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:53.612+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:53.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:53.612+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:53.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:53.612+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:53.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:53.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:53.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:53.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:53.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:53.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:54.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:54.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:54.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:54.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:54.547 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:54.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:54.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:54.658+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:54.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:54.658+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:54.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:54.658+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:54.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:54.557+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:54.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:54.557+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:54.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:54.557+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:55.604 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:55.314+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:55.604 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:55.314+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:55.604 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:55.314+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:55.604 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:55.557+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:55.605 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:55.557+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:55.605 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:55.557+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:55.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:55.614+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:55.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:55.614+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:55.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:55.614+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:56.591 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:56.556+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:56.591 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:56.556+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:56.591 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:56.556+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:56.592 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:56.316+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:56.592 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:56.316+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:56.592 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:56.316+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:56.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:56.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:56.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:56.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:56.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:56.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:57.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:01:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:01:56.869+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 52 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:01:57.521 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:57.290+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:57.522 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:57.290+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:57.522 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:57.290+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:57.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:57.611+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:57.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:57.611+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:57.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:57.611+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:57.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:57.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:57.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:57.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:57.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:57.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:58.631 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:58.492+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:58.632 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:58.492+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:58.632 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:58.492+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:58.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:58.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:58.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:58.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:58.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:58.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:58.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:58.642+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:58.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:58.642+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:58.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:58.642+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:59.605 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:59.513+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:59.606 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:59.513+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:59.606 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:01:59.513+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:01:59.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:59.277+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:59.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:59.277+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:59.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:01:59.277+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:01:59.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:59.616+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:59.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:59.616+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:01:59.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:01:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:01:59.616+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:00.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:00.489+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:00.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:00.489+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:00.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:00.489+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:00.560 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:00.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:00.560 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:00.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:00.561 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:00.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:00.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:00.570+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:00.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:00.570+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:00.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:00.570+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:01.551 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:01.471+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:01.552 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:01.471+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:01.552 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:01.471+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:01.552 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:01.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:01.552 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:01.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:01.552 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:01.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:01.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:01.562+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:01.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:01.562+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:01.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:01.562+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:02.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:01.869+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 70 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:02.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:02.455+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:02.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:02.455+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:02.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:02.455+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:02.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:02.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:02.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:02.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:02.584 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:02.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:02.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:02.593+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:02.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:02.593+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:02.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:02.593+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:03.218+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:03.218+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:03.218+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:03.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:03.452+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:03.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:03.452+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:03.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:03.452+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:03.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:03.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:03.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:03.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:03.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:03.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:04.613 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:04.251+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:04.613 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:04.251+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:04.613 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:04.251+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:04.613 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:04.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:04.614 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:04.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:04.614 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:04.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:04.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:04.623+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:04.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:04.623+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:04.962 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:04.623+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:05.638 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:05.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:05.638 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:05.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:05.638 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:05.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:05.639 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:05.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:05.639 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:05.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:05.639 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:05.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:05.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:05.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:05.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:05.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:05.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:05.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:06.629 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:06.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:06.630 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:06.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:06.630 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:06.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:06.630 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:06.312+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:06.630 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:06.312+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:06.630 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:06.312+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:06.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:06.869+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 86 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:06.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:06.639+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:06.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:06.639+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:06.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:06.639+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:07.595 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:07.442+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:07.596 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:07.442+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:07.596 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:07.442+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:07.596 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:07.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:07.596 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:07.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:07.596 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:07.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:07.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:07.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:07.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:07.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:07.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:07.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:08.550 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:08.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:08.550 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:08.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:08.550 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:08.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:08.551 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:08.410+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:08.551 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:08.410+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:08.551 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:08.410+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:08.814 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:08.560+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:08.815 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:08.560+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:08.815 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:08.560+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:09.543 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:09.391+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:09.544 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:09.391+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:09.544 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:09.391+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:09.544 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:09.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:09.544 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:09.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:09.544 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:09.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:09.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:09.553+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:09.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:09.553+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:09.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:09.553+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:10.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:10.528+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:10.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:10.528+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:10.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:10.528+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:10.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:10.422+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:10.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:10.422+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:10.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:10.422+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:10.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:10.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:10.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:10.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:10.711 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:10.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:11.564 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:11.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:11.565 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:11.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:11.565 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:11.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:11.565 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:11.470+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:11.565 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:11.470+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:11.565 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:11.470+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:11.860 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:11.574+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:11.860 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:11.574+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:11.861 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:11.574+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:12.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:11.870+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 104 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:12.538 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:12.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:12.539 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:12.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:12.539 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:12.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:12.540 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:12.429+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:12.540 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:12.429+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:12.540 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:12.429+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:12.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:12.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:12.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:12.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:12.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:12.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:13.503 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:13.242+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:13.503 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:13.242+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:13.504 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:13.242+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:13.504 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:13.422+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:13.504 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:13.422+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:13.505 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:13.422+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:13.866 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:13.513+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:13.867 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:13.513+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:13.867 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:13.513+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:14.540 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:14.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:14.540 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:14.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:14.540 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:14.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:14.541 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:14.423+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:14.541 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:14.423+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:14.541 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:14.423+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:14.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:14.550+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:14.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:14.550+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:14.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:14.550+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:15.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:15.228+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:15.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:15.228+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:15.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:15.228+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:15.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:15.425+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:15.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:15.425+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:15.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:15.425+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:15.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:15.592+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:15.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:15.592+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:15.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:15.592+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:16.572 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:16.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:16.572 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:16.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:16.572 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:16.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:16.572 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:16.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:16.572 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:16.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:16.573 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:16.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:16.860 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:16.581+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:16.861 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:16.581+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:16.861 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:16.581+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:17.181 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:16.870+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 123 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:17.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:17.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:17.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:17.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:17.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:17.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:17.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:17.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:17.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:17.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:17.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:17.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:17.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:17.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:17.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:18.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:18.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:18.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:18.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:18.386+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:18.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:18.386+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:18.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:18.386+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:18.912 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:18.594+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:18.912 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:18.594+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:18.913 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:18.594+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:19.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:19.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:19.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:19.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:19.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:19.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:19.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:19.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:19.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:19.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:19.632+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:19.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:19.632+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:19.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:19.632+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:20.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:20.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:20.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:20.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:20.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:20.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:20.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:20.388+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:20.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:20.388+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:20.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:20.388+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:20.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:20.653+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:20.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:20.653+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:20.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:20.653+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:21.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:21.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:21.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:21.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:21.383+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:21.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:21.383+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:21.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:21.383+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:21.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:21.871+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 142 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:21.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:21.691+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:21.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:21.691+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:21.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:21.691+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:22.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:22.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:22.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:22.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:22.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:22.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:22.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:22.348+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:22.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:22.348+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:22.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:22.348+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:22.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:22.662+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:22.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:22.662+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:22.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:22.662+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:23.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:23.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:23.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:23.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:23.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:23.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:23.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:23.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:23.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:23.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:23.654+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:23.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:23.654+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:23.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:23.654+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:24.365 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:24.119+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:24.366 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:24.119+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:24.366 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:24.119+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:24.666 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:24.376+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:24.666 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:24.376+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:24.666 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:24.376+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:24.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:24.677+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:24.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:24.677+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:24.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:24.677+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:25.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:25.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:25.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:25.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:25.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:25.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:25.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:25.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:25.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:25.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:25.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:25.144+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:25.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:25.144+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.1.8:152 2.11 2:8957603a:::201.00000007:head [write 2959636~2378 [fadvise_dontneed] in=2378b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:25.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:25.704+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:25.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:25.704+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:25.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:25.704+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:26.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:26.347+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:26.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:26.347+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:26.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:26.347+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:26.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:26.122+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:26.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:26.122+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:26.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:26.122+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:26.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:26.123+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.1.8:152 2.11 2:8957603a:::201.00000007:head [write 2959636~2378 [fadvise_dontneed] in=2378b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:26.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:26.871+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 158 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:26.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:26.697+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:26.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:26.697+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:26.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:26.697+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:27.364 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:27.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:27.364 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:27.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:27.364 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:27.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:27.365 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:27.103+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.1.8:152 2.11 2:8957603a:::201.00000007:head [write 2959636~2378 [fadvise_dontneed] in=2378b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:27.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:27.374+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:27.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:27.374+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:27.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:27.374+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:28.125 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:27.729+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:28.125 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:27.729+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:28.125 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:27.729+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:28.390 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:28.135+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:28.390 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:28.135+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:28.390 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:28.135+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:28.390 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:28.135+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.1.8:152 2.11 2:8957603a:::201.00000007:head [write 2959636~2378 [fadvise_dontneed] in=2378b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:28.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:28.400+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:28.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:28.400+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:28.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:28.400+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:29.010 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:28.735+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:29.010 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:28.735+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:29.010 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:28.735+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:29.388 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:29.134+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:29.389 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:29.134+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:29.389 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:29.134+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:29.389 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:29.134+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.1.8:152 2.11 2:8957603a:::201.00000007:head [write 2959636~2378 [fadvise_dontneed] in=2378b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:29.684 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:29.399+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:29.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:29.399+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:29.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:29.399+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:29.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:29.399+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:29.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:29.695+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:29.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:29.695+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:29.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:29.695+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:30.422 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:30.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:30.423 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:30.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:30.423 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:30.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:30.423 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:30.089+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:30.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:30.669+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:30.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:30.669+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:30.710 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:30.669+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:30.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:30.433+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:30.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:30.433+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:30.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:30.433+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:30.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:30.433+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:31.408 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:31.106+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:31.408 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:31.106+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:31.408 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:31.106+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:31.408 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:31.106+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:31.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:31.418+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:31.687 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:31.418+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:31.687 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:31.418+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:31.687 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:31.419+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:31.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:31.872+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 176 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:31.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:31.697+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:31.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:31.697+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:31.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:31.697+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:32.376 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:32.070+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:32.376 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:32.070+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:32.376 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:32.070+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:32.376 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:32.070+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:32.386+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:32.386+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:32.386+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:32.386+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:33.039 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:32.737+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:33.039 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:32.737+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:33.040 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:32.737+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:33.336 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:33.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:33.336 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:33.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:33.336 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:33.050+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:33.336 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:33.050+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:33.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:33.346+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:33.346+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:33.346+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:33.346+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:34.059 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:33.769+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:34.059 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:33.769+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:34.059 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:33.769+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:34.374 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:34.091+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:34.375 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:34.091+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:34.375 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:34.091+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:34.375 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:34.091+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:34.385+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:34.385+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:34.385+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:34.385+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:35.056 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:34.752+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:35.057 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:34.752+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:35.057 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:34.752+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:35.396 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:35.067+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:35.396 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:35.067+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:35.396 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:35.067+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:35.396 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:35.067+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:35.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:35.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:35.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:35.406+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:36.053 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:35.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:36.053 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:35.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:36.053 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:35.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:36.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:36.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:36.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:36.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:36.439 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:36.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:36.439 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:36.064+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:36.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:36.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:36.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:36.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:36.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:36.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:36.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:36.449+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:37.075 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:36.872+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 192 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:37.076 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:36.761+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:37.076 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:36.761+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:37.076 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:36.761+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:37.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:37.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:37.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:37.086+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:37.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:37.481+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:37.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:37.481+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:37.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:37.481+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:37.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:37.481+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:38.099 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:37.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:38.100 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:37.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:38.100 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:37.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:38.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:38.110+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:38.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:38.110+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:38.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:38.110+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:38.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:38.110+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:38.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:38.755+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:38.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:38.755+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:38.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:38.755+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:38.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:38.527+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:38.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:38.528+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:38.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:38.528+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:38.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:38.528+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:39.100+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:39.100+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:39.100+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:39.100+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:39.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:39.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:39.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:39.527+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:39.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:39.720+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:39.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:39.720+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:39.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:39.720+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:40.110+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:40.110+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:40.110+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:40.110+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:40.744+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:40.744+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:40.744+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:40.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:40.537+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:40.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:40.537+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:40.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:40.537+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:40.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:40.537+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:41.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:41.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:41.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:41.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:41.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:41.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:41.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:41.117+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:41.863 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:41.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:41.863 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:41.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:41.864 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:41.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:41.864 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:41.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:41.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:41.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:41.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:41.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:41.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:41.544+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:42.122 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:41.873+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 210 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:42.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:42.133+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:42.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:42.133+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:42.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:42.133+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:42.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:42.133+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:42.805+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:42.805+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:42.805+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:42.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:42.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:42.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:42.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:42.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:42.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:42.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:42.572+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:43.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:43.123+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:43.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:43.123+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:43.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:43.123+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:43.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:43.123+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:43.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:43.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:43.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:43.570+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:43.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:43.810+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:43.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:43.810+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:43.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:43.810+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:44.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:44.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:44.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:44.086+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:44.839 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:44.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:44.839 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:44.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:44.839 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:44.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:44.839 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:44.569+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:45.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:45.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:45.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:45.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:45.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:45.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:45.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:45.042+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:45.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:44.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:45.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:44.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:45.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:44.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:45.877 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:45.615+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:45.877 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:45.615+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:45.878 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:45.615+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:45.878 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:45.615+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:46.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:46.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:46.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:46.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:46.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:46.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:46.041+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:46.213 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:45.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:46.213 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:45.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:46.213 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:45.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:46.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:46.873+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 226 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:46.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:46.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:46.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:46.881+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:46.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:46.881+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:46.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:46.643+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:46.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:46.643+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:46.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:46.643+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:46.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:46.643+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:47.080+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:47.080+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:47.080+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:47.081+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:47.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:47.858+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:47.858+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:47.858+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:47.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:47.655+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:47.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:47.655+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:47.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:47.655+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:47.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:47.655+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:48.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:48.071+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:48.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:48.071+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:48.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:48.071+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:48.465 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:48.071+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:48.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:48.902+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:48.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:48.902+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:48.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:48.902+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:48.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:48.689+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:48.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:48.689+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:48.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:48.689+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:48.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:48.689+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:49.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:49.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:49.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:49.061+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:49.930 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:49.643+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:49.930 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:49.643+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:49.930 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:49.643+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:49.930 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:49.643+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:50.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:49.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:50.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:49.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:50.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:49.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:50.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:50.043+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:50.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:50.043+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:50.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:50.043+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:50.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:50.043+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:50.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:50.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:50.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:50.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:50.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:50.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:50.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:50.604+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:51.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:50.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:51.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:50.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:51.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:50.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:51.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:51.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:51.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:51.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:51.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:51.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:51.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:51.064+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:51.841 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:51.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:51.841 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:51.600+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:51.842 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:51.600+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:51.842 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:51.600+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:52.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:51.874+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 244 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:52.227 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:51.852+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:52.228 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:51.852+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:52.228 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:51.852+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:52.229 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:52.101+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:52.229 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:52.101+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:52.229 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:52.101+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:52.229 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:52.101+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:52.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:52.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:52.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:52.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:52.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:52.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:52.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:52.584+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:52.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:52.584+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:52.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:52.584+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:52.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:52.584+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:53.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:53.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:53.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:53.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:53.088+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:53.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:53.793+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:53.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:53.793+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:53.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:53.793+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:53.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:53.594+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:53.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:53.594+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:53.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:53.594+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:53.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:53.594+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:54.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:54.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:54.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:54.127+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:54.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:54.796+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:54.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:54.796+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:54.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:54.796+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:54.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:54.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:54.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:54.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:54.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:54.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:54.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:54.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:55.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:55.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:55.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:55.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:55.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:55.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:55.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:55.153+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:55.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:55.765+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:55.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:55.765+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:55.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:55.765+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:55.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:55.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:55.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:55.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:55.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:55.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:55.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:55.587+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:56.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:56.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:56.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:56.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:56.182+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:56.864 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:56.606+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:56.864 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:56.606+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:56.864 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:56.606+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:56.864 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:56.606+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:56.864 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:56.793+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:56.865 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:56.793+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:56.865 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:56.793+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:57.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:02:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:02:56.873+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 261 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:02:57.562 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:57.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:57.563 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:57.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:57.563 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:57.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:57.563 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:57.230+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:57.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:57.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:57.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:57.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:57.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:57.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:57.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:57.572+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:58.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:57.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:58.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:57.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:58.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:57.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:58.553 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:58.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:58.554 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:58.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:58.554 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:58.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:58.554 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:58.243+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:58.843 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:58.564+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:58.843 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:58.564+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:58.843 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:58.564+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:58.844 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:58.564+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:59.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:58.853+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:59.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:58.853+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:59.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:58.853+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:02:59.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:59.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:59.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:59.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:59.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:59.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:02:59.546 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:02:59.237+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:02:59.807 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:59.555+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:59.807 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:59.555+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:59.807 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:59.555+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:02:59.807 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:02:59.555+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:00.179 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:59.817+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:00.179 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:59.817+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:00.179 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:02:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:02:59.817+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:00.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:00.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:00.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:00.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:00.189+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:00.797 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:00.525+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:00.798 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:00.525+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:00.798 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:00.525+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:00.798 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:00.525+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:01.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:00.807+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:01.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:00.807+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:01.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:00.807+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:01.559 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:01.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:01.559 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:01.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:01.560 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:01.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:01.560 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:01.236+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:01.830 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:01.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:01.831 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:01.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:01.831 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:01.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:01.831 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:01.569+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:02.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:01.874+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 279 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:02.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:01.841+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:02.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:01.841+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:02.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:01.841+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:02.600 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:02.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:02.600 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:02.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:02.600 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:02.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:02.600 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:02.241+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:02.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:02.810+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:02.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:02.810+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:02.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:02.810+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:02.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:02.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:02.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:02.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:02.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:02.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:02.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:02.610+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:03.625 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:03.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:03.625 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:03.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:03.625 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:03.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:03.625 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:03.256+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:03.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:03.839+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:03.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:03.839+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:03.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:03.839+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:03.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:03.635+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:03.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:03.635+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:03.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:03.635+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:03.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:03.635+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:04.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:04.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:04.664 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:04.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:04.664 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:04.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:04.664 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:04.267+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:04.922 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:04.674+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:04.922 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:04.674+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:04.922 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:04.674+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:04.922 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:04.674+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:04.923 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:04.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:04.923 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:04.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:04.923 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:04.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:05.654 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:05.281+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:05.655 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:05.281+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:05.655 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:05.281+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:05.655 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:05.281+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:05.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:05.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:05.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:05.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:05.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:05.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:05.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:05.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:05.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:05.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:05.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:05.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:05.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:05.665+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:06.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:06.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:06.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:06.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:06.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:06.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:06.620 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:06.270+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:06.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:06.874+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 295 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:06.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:06.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:06.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:06.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:06.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:06.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:06.630+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:06.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:06.630+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:06.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:06.630+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:06.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:06.630+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:07.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:07.248+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:07.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:07.248+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:07.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:07.248+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:07.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:07.248+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:07.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:07.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:07.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:07.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:07.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:07.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:07.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:07.629+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:07.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:07.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:07.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:07.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:07.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:07.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:08.594 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:08.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:08.594 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:08.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:08.595 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:08.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:08.595 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:08.261+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:08.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:08.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:08.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:08.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:08.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:08.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:08.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:08.604+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:08.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:08.814+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:08.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:08.814+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:08.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:08.814+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:09.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:09.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:09.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:09.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:09.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:09.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:09.582 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:09.263+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:09.839 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:09.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:09.839 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:09.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:09.840 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:09.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:09.840 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:09.592+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:10.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:09.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:10.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:09.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:10.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:09.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:10.569 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:10.252+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:10.569 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:10.252+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:10.569 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:10.252+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:10.569 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:10.252+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:10.881 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:10.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:10.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:10.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:10.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:10.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:10.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:10.579+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:11.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:10.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:11.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:10.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:11.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:10.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:11.521 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:11.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:11.522 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:11.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:11.522 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:11.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:11.522 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:11.272+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:11.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:11.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:11.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:11.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:11.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:11.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:11.865 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:11.531+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:12.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:11.875+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 313 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:12.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:11.883+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:12.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:11.883+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:12.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:11.883+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:12.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:12.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:12.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:12.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:12.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:12.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:12.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:12.292+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:12.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:12.498+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:12.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:12.498+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:12.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:12.498+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:12.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:12.498+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:13.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:12.861+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:13.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:12.861+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:13.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:12.861+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:13.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:13.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:13.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:13.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:13.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:13.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:13.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:13.315+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:13.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:13.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:13.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:13.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:13.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:13.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:13.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:13.531+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:14.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:13.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:14.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:13.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:14.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:13.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:14.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:14.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:14.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:14.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:14.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:14.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:14.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:14.313+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:14.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:14.542+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:14.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:14.542+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:14.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:14.542+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:14.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:14.542+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:15.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:14.816+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:15.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:14.816+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:15.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:14.816+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:15.515 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:15.268+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:15.515 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:15.268+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:15.515 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:15.268+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:15.515 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:15.268+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:15.776 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:15.525+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:15.777 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:15.525+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:15.777 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:15.525+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:15.777 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:15.525+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:16.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:15.787+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:16.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:15.787+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:16.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:15.787+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:16.529 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:16.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:16.529 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:16.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:16.529 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:16.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:16.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:16.282+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:16.783 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:16.540+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:16.783 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:16.540+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:16.784 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:16.540+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:16.784 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:16.540+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:17.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:16.875+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 329 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:17.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:16.794+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:17.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:16.794+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:17.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:16.794+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:17.579 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:17.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:17.579 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:17.234+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:17.579 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:17.234+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:17.579 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:17.234+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:17.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:17.771+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:17.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:17.771+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:17.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:17.771+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:17.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:17.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:17.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:17.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:17.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:17.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:17.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:17.590+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:18.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:18.194+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:18.194+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:18.194+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:18.194+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:18.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:18.623+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:18.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:18.623+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:18.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:18.623+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:18.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:18.623+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:18.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:18.806+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:18.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:18.806+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:18.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:18.806+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:19.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:19.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:19.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:19.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:19.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:19.180+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:19.788 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:19.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:19.788 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:19.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:19.788 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:19.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:19.788 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:19.638+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:20.130 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:19.799+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:20.130 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:19.799+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:20.130 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:19.799+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:20.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:20.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:20.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:20.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:20.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:20.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:20.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:20.141+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:20.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:20.815+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:20.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:20.815+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:20.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:20.815+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:20.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:20.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:20.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:20.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:20.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:20.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:20.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:20.682+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:21.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:21.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:21.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:21.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:21.128+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:21.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:21.877+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 347 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:21.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:21.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:21.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:21.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:21.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:21.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:21.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:21.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:21.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:21.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:21.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:21.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:21.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:21.703+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:22.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:22.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:22.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:22.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:22.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:22.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:22.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:22.162+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:22.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:22.890+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:22.962 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:22.890+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:22.962 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:22.890+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:22.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:22.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:22.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:22.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:22.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:22.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:22.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:22.703+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:23.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:23.117+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:23.117+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:23.117+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:23.117+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:23.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:23.870+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:23.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:23.870+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:23.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:23.870+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:23.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:23.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:23.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:23.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:23.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:23.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:23.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:23.699+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:24.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:24.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:24.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:24.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:24.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:24.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:24.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:24.151+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:24.899 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:24.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:24.899 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:24.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:24.899 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:24.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:24.899 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:24.732+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:25.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:24.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:25.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:24.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:25.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:24.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:25.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:25.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:25.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:25.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:25.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:25.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:25.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:25.116+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:26.139 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:25.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:26.139 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:25.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:26.139 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:25.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:26.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:25.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:26.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:25.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:26.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:25.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:26.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:25.725+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:26.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:26.149+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:26.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:26.149+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:26.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:26.149+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:26.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:26.150+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:26.954 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:26.877+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 368 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:26.954 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:26.743+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:26.954 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:26.743+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:26.955 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:26.743+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:26.955 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:26.744+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:27.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:26.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:27.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:26.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:27.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:26.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:27.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:27.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:27.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:27.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:27.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:27.114+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:27.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:27.115+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:28.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:27.767+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:28.138 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:27.767+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:28.138 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:27.767+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:28.138 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:27.767+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:28.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:27.959+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:28.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:27.959+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:28.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:27.959+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:28.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:28.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:28.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:28.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:28.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:28.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:28.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:28.148+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:29.141 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:28.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:29.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:28.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:29.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:28.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:29.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:28.783+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:29.142 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:28.949+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:29.142 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:28.949+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:29.143 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:28.949+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:29.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:29.152+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:29.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:29.152+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:29.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:29.152+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:29.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:29.152+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:29.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:29.777+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:29.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:29.777+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:29.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:29.777+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:29.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:29.777+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:30.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:30.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:30.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:30.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:30.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:30.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:30.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:30.189+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:30.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:29.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:30.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:29.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:30.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:29.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:31.174 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:31.016+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:31.174 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:31.016+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:31.175 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:31.016+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:31.175 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:30.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:31.175 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:30.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:31.175 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:30.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:31.175 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:30.798+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:31.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:31.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:31.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:31.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:31.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:31.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:31.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:31.185+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:32.152 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:31.878+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 386 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:32.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:31.986+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:32.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:31.986+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:32.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:31.986+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:32.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:31.806+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:32.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:31.806+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:32.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:31.806+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:32.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:31.806+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:32.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:32.163+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:32.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:32.163+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:32.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:32.163+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:32.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:32.163+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:33.180 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:32.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:33.181 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:32.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:33.181 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:32.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:33.183 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:32.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:33.183 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:32.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:33.183 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:32.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:33.183 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:32.758+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:33.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:33.190+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:33.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:33.190+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:33.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:33.190+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:33.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:33.190+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:34.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:33.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:34.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:33.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:34.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:33.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:34.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:33.786+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:34.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:33.786+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:34.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:33.786+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:34.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:33.786+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:34.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:34.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:34.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:34.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:34.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:34.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:34.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:34.200+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:34.900 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:34.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:34.900 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:34.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:34.901 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:34.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:34.901 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:34.796+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:35.151 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:34.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:35.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:34.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:35.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:34.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:35.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:35.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:35.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:35.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:35.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:35.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:35.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:35.163+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:36.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:35.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:36.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:35.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:36.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:35.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:36.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:35.787+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:36.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:35.787+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:36.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:35.787+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:36.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:35.787+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:36.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:36.198+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:36.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:36.198+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:36.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:36.198+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:36.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:36.198+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:37.187 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:36.878+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 402 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:37.188 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:36.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:37.188 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:36.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:37.188 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:36.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:37.189 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:36.767+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:37.189 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:36.767+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:37.189 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:36.767+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:37.189 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:36.767+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:37.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:37.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:37.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:37.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:37.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:37.199+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:38.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:37.852+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:38.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:37.852+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:38.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:37.852+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:38.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:37.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:38.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:37.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:38.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:37.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:38.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:37.814+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:38.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:38.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:38.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:38.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:38.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:38.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:38.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:38.238+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:39.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:38.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:39.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:38.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:39.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:38.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:39.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:38.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:39.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:38.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:39.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:38.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:39.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:38.765+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:39.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:39.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:39.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:39.201+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:39.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:39.829+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:39.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:39.829+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:39.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:39.829+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:39.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:39.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:39.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:39.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:39.741+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:40.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:40.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:40.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:40.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:40.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:40.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:40.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:40.229+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:40.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:40.851+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:40.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:40.851+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:40.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:40.851+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:40.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:40.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:40.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:40.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:40.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:40.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:40.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:40.692+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:41.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:41.258+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:41.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:41.258+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:41.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:41.258+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:41.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:41.258+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:41.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:41.879+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 420 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:41.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:41.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:41.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:41.714+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:41.714+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:41.714+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:41.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:41.714+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:42.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:42.293+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:42.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:42.293+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:42.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:42.293+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:42.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:42.293+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:42.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:42.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:42.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:42.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:42.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:42.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:42.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:42.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:42.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:42.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:42.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:42.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:42.683+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:43.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:43.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:43.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:43.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:43.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:43.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:43.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:43.335+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:43.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:43.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:43.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:43.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:43.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:43.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:43.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:43.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:43.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:43.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:43.683+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:44.691 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:44.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:44.692 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:44.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:44.692 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:44.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:44.692 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:44.300+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:44.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:44.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:44.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:44.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:44.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:44.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:44.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:44.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:44.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:44.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:44.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:44.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:44.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:44.703+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:45.665 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:45.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:45.665 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:45.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:45.665 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:45.287+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:45.665 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:45.287+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:45.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:45.890+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:45.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:45.890+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:45.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:45.890+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:45.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:45.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:45.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:45.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:45.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:45.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:45.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:45.676+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:46.674 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:46.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:46.674 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:46.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:46.674 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:46.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:46.674 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:46.254+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:46.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:46.878+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 441 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:46.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:46.897+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:46.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:46.897+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:46.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:46.897+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:46.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:46.685+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:46.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:46.685+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:46.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:46.685+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:46.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:46.685+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:47.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:47.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:47.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:47.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:47.211+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:47.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:47.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:47.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:47.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:47.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:47.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:47.696+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:47.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:47.696+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:47.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:47.696+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:47.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:47.696+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:48.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:48.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:48.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:48.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:48.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:48.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:48.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:48.243+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:49.194 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:48.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:49.195 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:48.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:49.195 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:48.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:49.195 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:48.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:49.195 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:48.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:49.195 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:48.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:49.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:48.740+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:49.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:49.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:49.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:49.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:49.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:49.205+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:50.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:49.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:50.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:49.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:50.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:49.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:50.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:49.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:50.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:49.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:50.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:49.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:50.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:49.755+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:50.697 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:50.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:50.698 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:50.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:50.698 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:50.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:50.698 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:50.239+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:50.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:50.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:50.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:50.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:50.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:50.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:50.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:50.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:50.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:50.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:50.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:50.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:50.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:50.707+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:51.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:51.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:51.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:51.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:51.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:51.191+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:51.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:51.879+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 462 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:51.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:51.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:51.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:51.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:51.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:51.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:51.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:51.692+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:52.651 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:52.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:52.652 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:52.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:52.652 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:52.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:52.652 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:52.233+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:52.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:52.662+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:52.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:52.662+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:52.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:52.662+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:52.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:52.662+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:53.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:52.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:53.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:52.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:53.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:52.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:53.654 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:53.223+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:53.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:53.223+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:53.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:53.223+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:53.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:53.223+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:53.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:53.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:53.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:53.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:53.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:53.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:53.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:53.665+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:54.251 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:54.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:54.252 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:54.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:54.252 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:54.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:54.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:54.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:54.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:54.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:54.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:54.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:54.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:54.262+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:54.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:54.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:54.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:54.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:54.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:54.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:54.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:54.642+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:55.285 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:54.970+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:55.286 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:54.970+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:55.286 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:54.970+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:55.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:55.296+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:55.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:55.296+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:55.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:55.296+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:55.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:55.296+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:55.930 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:55.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:55.930 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:55.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:55.931 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:55.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:55.931 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:55.642+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:56.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:55.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:56.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:55.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:56.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:55.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:56.610 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:56.284+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:56.610 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:56.284+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:56.610 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:56.284+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:56.610 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:56.284+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:56.869 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:56.620+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:56.870 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:56.620+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:56.870 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:56.620+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:56.870 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:56.620+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:57.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:03:56.880+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 478 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:03:57.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:56.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:57.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:56.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:57.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:56.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:57.574 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:57.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:57.574 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:57.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:57.574 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:57.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:57.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:57.297+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:57.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:57.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:57.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:57.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:57.585+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:58.238 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:57.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:58.239 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:57.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:58.239 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:57.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:58.590 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:58.249+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:58.591 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:58.249+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:58.591 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:58.249+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:58.591 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:58.249+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:58.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:58.601+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:58.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:58.601+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:58.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:58.601+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:58.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:58.601+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:59.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:59.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:59.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:59.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:59.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:03:59.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:03:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:59.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:59.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:59.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:03:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:03:59.267+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:03:59.892 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:59.622+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:59.892 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:59.622+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:59.892 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:59.622+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:03:59.892 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:03:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:03:59.622+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:00.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:00.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:00.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:00.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:00.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:00.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:00.622 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:00.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:00.622 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:00.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:00.622 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:00.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:00.622 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:00.243+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:00.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:00.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:00.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:00.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:00.633+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:01.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:01.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:01.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:01.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:01.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:01.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:01.210+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:01.210+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:01.210+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:01.210+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:01.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:01.880+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 496 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:01.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:01.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:01.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:01.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:01.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:01.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:01.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:01.633+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:02.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:02.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:02.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:02.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:02.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:02.243+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:02.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:02.244+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:02.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:02.075+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:02.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:02.075+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:02.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:02.075+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:02.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:02.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:02.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:02.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:02.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:02.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:02.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:02.665+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:03.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:03.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:03.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:03.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:03.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:03.269+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:03.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:03.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:03.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:03.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:03.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:03.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:03.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:03.624+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:03.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:03.624+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:03.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:03.624+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:03.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:03.624+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:04.304 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:04.058+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:04.304 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:04.058+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:04.304 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:04.058+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:04.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:04.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:04.631 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:04.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:04.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:04.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:04.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:04.315+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:04.941 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:04.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:04.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:04.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:04.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:04.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:04.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:04.642+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:05.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:05.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:05.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:05.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:05.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:05.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:05.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:05.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:05.668 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:05.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:05.668 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:05.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:05.668 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:05.275+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:05.678+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:05.678+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:05.678+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:05.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:05.678+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:06.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:06.061+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:06.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:06.061+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:06.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:06.061+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:06.249+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:06.249+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:06.249+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:06.249+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:06.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:06.881+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 512 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:06.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:06.717+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:06.717+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:06.717+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:06.717+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:07.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:07.259+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:07.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:07.259+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:07.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:07.259+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:07.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:07.259+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:07.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:07.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:07.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:07.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:07.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:07.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:08.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:07.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:08.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:07.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:08.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:07.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:08.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:07.747+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:08.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:08.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:08.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:08.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:08.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:08.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:08.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:08.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:08.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:08.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:08.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:08.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:08.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:08.261+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:08.990 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:08.735+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:08.990 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:08.735+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:08.990 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:08.735+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:08.990 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:08.735+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:09.253 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:09.001+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:09.254 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:09.001+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:09.254 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:09.001+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:09.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:09.264+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:09.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:09.264+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:09.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:09.264+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:09.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:09.265+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:09.990 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:09.975+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:09.991 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:09.975+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:09.991 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:09.975+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:09.991 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:09.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:09.991 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:09.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:09.991 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:09.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:09.991 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:09.732+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:10.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:10.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:10.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:10.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:10.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:10.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:10.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:10.248+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:10.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:10.718+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:10.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:10.718+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:10.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:10.718+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:10.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:10.718+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:11.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:11.213+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:11.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:11.213+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:11.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:11.213+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:11.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:11.213+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:11.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:11.007+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:11.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:11.007+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:11.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:11.007+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:12.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:11.882+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 530 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:12.200 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:11.983+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:12.200 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:11.983+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:12.200 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:11.983+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:12.201 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:11.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:12.201 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:11.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:12.201 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:11.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:12.201 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:11.747+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:12.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:12.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:12.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:12.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:12.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:12.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:12.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:12.211+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:13.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:12.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:13.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:12.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:13.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:12.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:13.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:12.774+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:13.194 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:13.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:13.195 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:13.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:13.195 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:13.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:13.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:13.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:13.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:13.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:13.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:13.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:13.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:13.205+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:14.017 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:13.753+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:14.017 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:13.753+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:14.017 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:13.753+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:14.017 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:13.753+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:14.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:14.027+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:14.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:14.027+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:14.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:14.027+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:14.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:14.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:14.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:14.161+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:15.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:14.789+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:15.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:14.789+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:15.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:14.789+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:15.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:14.789+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:15.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:15.047+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:15.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:15.047+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:15.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:15.047+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:15.167+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:15.167+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:15.167+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:15.168+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:16.044 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:15.744+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:16.044 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:15.744+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:16.044 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:15.744+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:16.045 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:15.745+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:16.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:16.055+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:16.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:16.055+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:16.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:16.055+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:16.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:16.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:16.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:16.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:16.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:16.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:16.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:16.191+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:17.004 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:16.882+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 546 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:17.005 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:16.739+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:17.005 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:16.739+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:17.005 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:16.739+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:17.005 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:16.739+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:17.305 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:17.015+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:17.306 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:17.015+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:17.306 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:17.015+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:17.306 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:17.163+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:17.306 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:17.163+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:17.306 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:17.163+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:17.307 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:17.163+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:18.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:17.985+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:18.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:17.985+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:18.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:17.985+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:18.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:17.772+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:18.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:17.772+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:18.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:17.772+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:18.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:17.772+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:18.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:18.149+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:18.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:18.149+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:18.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:18.149+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:18.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:18.149+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:19.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:19.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:19.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:18.768+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:18.768+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:18.768+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:19.091 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:18.768+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:19.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:19.102+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:19.102+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:19.102+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:19.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:19.102+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:20.088 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:20.032+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:20.088 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:20.032+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:20.089 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:20.032+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:20.089 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:19.801+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:20.089 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:19.801+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:20.089 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:19.801+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:20.089 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:19.801+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:20.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:20.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:20.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:20.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:20.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:20.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:20.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:20.151+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:20.964 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-04-15T23:04:21.121 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:21.018+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:21.121 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:21.018+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:21.121 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:21.018+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:21.121 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:20.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:21.121 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:20.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:21.122 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:20.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:21.122 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:20.783+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:21.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:21.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:21.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:21.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:21.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:21.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:21.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:21.132+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:22.136 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:21.883+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 594 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:22.136 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:22.049+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:22.136 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:22.049+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:22.136 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:22.049+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:22.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:21.815+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:22.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:21.815+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:22.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:21.815+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:22.137 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:21.815+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:22.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:22.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:22.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:22.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:22.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:22.147+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:22.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:22.147+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:22.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:22.859+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:22.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:22.859+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:22.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:22.859+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:22.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:22.859+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:23.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:23.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:23.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:23.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:23.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:23.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:23.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:23.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:23.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:23.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:23.166+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:24.184 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:24.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:24.185 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:24.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:24.185 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:24.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:24.185 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:23.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:24.185 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:23.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:24.185 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:23.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:24.186 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:23.888+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:24.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:24.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:24.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:24.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:24.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:24.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:24.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:24.196+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:25.119 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-04-15T23:04:25.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:24.983+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:25.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:24.983+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:25.138 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:24.983+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:25.138 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:24.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:25.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:24.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:25.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:24.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:25.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:24.881+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:25.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:25.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:25.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:25.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:25.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:25.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:25.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:25.238+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:26.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:25.994+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:26.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:25.994+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:26.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:25.994+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:26.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:25.913+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:26.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:25.913+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:26.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:25.913+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:26.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:25.913+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:26.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:26.280+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:26.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:26.280+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:26.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:26.280+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:26.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:26.281+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:27.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:26.883+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 643 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:27.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:27.017+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:27.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:27.017+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:27.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:27.017+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:27.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:26.923+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:27.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:26.923+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:27.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:26.923+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:27.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:26.923+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:27.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:27.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:27.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:27.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:27.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:27.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:27.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:27.265+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:28.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:28.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:28.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:28.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:28.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:28.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:28.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:27.890+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:28.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:27.890+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:28.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:27.890+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:28.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:27.890+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:28.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:28.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:28.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:28.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:28.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:28.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:28.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:28.270+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:29.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:29.048+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:29.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:29.048+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:29.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:29.048+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:29.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:28.877+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:28.877+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:28.877+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:28.877+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:29.657 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:29.311+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:29.658 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:29.311+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:29.658 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:29.311+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:29.658 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:29.311+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:30.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:30.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:30.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:30.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:30.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:30.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:30.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:29.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:30.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:29.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:30.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:29.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:30.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:29.866+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:30.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:30.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:30.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:30.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:30.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:30.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:30.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:30.300+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:31.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:31.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:31.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:31.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:31.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:31.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:31.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:30.840+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:31.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:30.840+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:31.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:30.840+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:31.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:30.840+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:31.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:31.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:31.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:31.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:31.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:31.270+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:31.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:31.270+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:32.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:31.884+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 700 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:32.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:31.857+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:32.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:31.857+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:32.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:31.857+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:32.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:31.858+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:32.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:32.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:32.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:32.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:32.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:32.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:32.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:32.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:32.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:32.279+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:33.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:32.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:33.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:32.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:33.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:32.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:33.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:32.823+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:33.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:33.049+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:33.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:33.049+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:33.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:33.049+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:33.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:33.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:33.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:33.301+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:34.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:33.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:34.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:33.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:34.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:33.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:34.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:33.849+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:34.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:34.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:34.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:34.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:34.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:34.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:34.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:34.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:34.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:34.275+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:35.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:34.835+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:35.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:34.835+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:35.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:34.835+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:35.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:34.835+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:35.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:35.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:35.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:35.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:35.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:35.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:35.231+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:35.231+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:35.231+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:35.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:35.231+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:36.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:36.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:36.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:36.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:36.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:36.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:35.859+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:36.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:35.859+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:36.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:35.859+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:36.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:35.859+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:36.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:36.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:36.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:36.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:36.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:36.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:36.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:36.220+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:37.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:36.884+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 757 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:37.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:37.059+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:37.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:37.059+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:37.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:37.059+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:37.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:36.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:37.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:36.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:37.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:36.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:37.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:36.894+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:37.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:37.241+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:37.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:37.241+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:37.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:37.241+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:37.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:37.241+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:38.186 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:38.083+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:38.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:38.083+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:38.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:38.083+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:38.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:37.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:38.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:37.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:38.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:37.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:38.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:37.858+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:38.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:38.197+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:38.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:38.197+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:38.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:38.197+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:38.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:38.198+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:39.158 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:39.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:39.159 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:39.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:39.159 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:39.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:39.159 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:38.902+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:39.159 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:38.902+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:39.159 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:38.902+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:39.159 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:38.902+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:39.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:39.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:39.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:39.169+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:40.169 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:39.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:40.170 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:39.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:40.170 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:39.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:40.170 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:39.886+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:40.170 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:40.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:40.171 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:40.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:40.171 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:40.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:40.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:40.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:40.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:40.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:40.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:40.180+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:41.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:41.021+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:41.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:41.021+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:41.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:41.021+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:41.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:40.878+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:41.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:40.878+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:41.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:40.878+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:41.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:40.878+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:41.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:41.223+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:41.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:41.223+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:41.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:41.223+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:41.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:41.223+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:42.210 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:41.884+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 816 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:41.891+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:41.891+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:41.891+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:41.891+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:41.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:41.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:42.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:41.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:42.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:42.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:42.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:42.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:42.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:42.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:42.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:42.254+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:43.197 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:43.017+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:43.198 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:43.017+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:43.198 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:43.017+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:43.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:42.851+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:43.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:42.851+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:43.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:42.851+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:43.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:42.851+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:43.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:43.207+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:43.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:43.207+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:43.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:43.207+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:43.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:43.208+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:44.176 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:44.037+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:44.176 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:44.037+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:44.177 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:44.037+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:44.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:43.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:44.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:43.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:44.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:43.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:44.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:43.849+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:44.186+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:44.186+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:44.186+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:44.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:44.186+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:45.149 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:45.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:45.149 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:45.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:45.149 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:45.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:45.149 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:44.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:45.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:44.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:45.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:44.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:45.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:44.814+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:45.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:45.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:45.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:45.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:45.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:45.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:45.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:45.160+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:46.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:45.807+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:46.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:45.807+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:46.102 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:45.807+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:46.102 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:45.807+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:46.102 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:46.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:46.102 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:46.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:46.102 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:46.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:46.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:46.111+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:46.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:46.111+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:46.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:46.111+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:46.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:46.111+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:47.079 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:46.885+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 873 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:47.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:46.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:47.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:46.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:47.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:46.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:47.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:46.790+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:47.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:47.089+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:47.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:47.089+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:47.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:47.089+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:47.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:47.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:47.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:47.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:47.128+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:48.123 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:47.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:48.123 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:47.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:48.124 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:47.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:48.124 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:47.783+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:48.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:48.134+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:48.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:48.134+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:48.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:48.134+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:48.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:48.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:48.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:48.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:48.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:48.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:48.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:48.156+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:49.133 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:48.810+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:49.133 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:48.810+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:49.133 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:48.810+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:49.133 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:48.810+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:49.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:49.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:49.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:49.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:49.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:49.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:49.177+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:49.177+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:49.177+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:49.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:49.177+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:50.106 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:49.804+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:50.106 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:49.804+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:50.106 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:49.804+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:50.107 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:49.804+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:50.383 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:50.117+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:50.384 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:50.117+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:50.384 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:50.117+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:50.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:50.212+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:50.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:50.212+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:50.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:50.212+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:50.384 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:50.212+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:51.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:50.786+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:51.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:50.786+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:51.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:50.786+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:51.140 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:50.786+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:51.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:51.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:51.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:51.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:51.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:51.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:51.224+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:51.224+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:51.224+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:51.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:51.224+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:52.176 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:51.885+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 932 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:52.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:51.764+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:52.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:51.764+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:52.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:51.764+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:52.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:51.764+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:52.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:52.187+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:52.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:52.187+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:52.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:52.187+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:52.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:52.217+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:52.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:52.217+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:52.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:52.217+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:52.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:52.217+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:52.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:52.717+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:52.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:52.717+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:52.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:52.717+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:52.962 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:52.717+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:53.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:53.157+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:53.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:53.157+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:53.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:53.157+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:53.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:53.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:53.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:53.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:53.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:53.222+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:54.160 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:53.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:54.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:53.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:54.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:53.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:54.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:53.721+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:54.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:54.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:54.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:54.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:54.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:54.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:54.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:54.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:54.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:54.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:54.236+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:55.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:54.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:55.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:54.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:55.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:54.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:55.161 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:54.724+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:55.432 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:55.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:55.433 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:55.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:55.433 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:55.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:55.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:55.186+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:55.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:55.186+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:55.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:55.186+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:55.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:55.187+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:55.712+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:55.712+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:55.712+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:55.712+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:56.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:56.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:56.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:56.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:56.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:56.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:56.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:56.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:56.233+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:56.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:56.233+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:57.152 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:04:56.885+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 989 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:04:57.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:56.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:57.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:56.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:57.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:56.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:57.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:56.741+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:57.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:57.163+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:57.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:57.163+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:57.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:57.163+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:57.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:57.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:57.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:57.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:57.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:57.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:57.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:57.194+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:58.126 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:57.782+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:58.126 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:57.782+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:58.126 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:57.782+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:58.126 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:57.782+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:58.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:58.137+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:58.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:58.137+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:58.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:58.137+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:58.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:58.217+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:58.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:58.217+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:58.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:58.217+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:58.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:58.217+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:59.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:58.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:59.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:58.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:59.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:58.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:04:59.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:58.733+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:04:59.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:59.092+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:59.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:59.092+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:59.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:04:59.092+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:04:59.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:59.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:59.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:59.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:04:59.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:04:59.169+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:00.076 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:59.750+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:00.076 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:59.750+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:00.077 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:59.750+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:00.077 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:04:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:04:59.750+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:00.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:00.087+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:00.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:00.087+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:00.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:00.087+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:00.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:00.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:00.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:00.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:00.128+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:00.739+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:00.739+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:00.739+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:00.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:00.739+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:01.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:01.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:01.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:01.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:01.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:01.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:01.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:01.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:01.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:01.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:01.160+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:02.042 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:01.887+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1048 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:02.042 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:01.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:02.042 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:01.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:02.043 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:01.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:02.043 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:01.740+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:02.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:02.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:02.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:02.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:02.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:02.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:02.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:02.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:02.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:02.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:02.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:02.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:02.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:02.141+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:03.011 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:02.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:03.012 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:02.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:03.012 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:02.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:03.012 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:02.773+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:03.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:03.023+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:03.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:03.023+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:03.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:03.023+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:03.138+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:03.138+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:03.138+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:03.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:03.138+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:04.085 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:04.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:04.085 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:04.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:04.086 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:04.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:04.086 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:03.787+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:04.086 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:03.787+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:04.086 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:03.787+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:04.086 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:03.788+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:04.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:04.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:04.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:04.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:04.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:04.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:04.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:04.097+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:05.013 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:04.753+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:05.014 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:04.753+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:05.014 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:04.753+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:05.017 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:04.754+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:05.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:05.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:05.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:05.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:05.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:05.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:05.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:05.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:05.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:05.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:05.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:05.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:05.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:05.094+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:05.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:05.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:05.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:05.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:05.766+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:06.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:06.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:06.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:06.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:06.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:06.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:06.085+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:06.085+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:06.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:06.085+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:06.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:06.085+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:07.069 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:06.888+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1105 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:07.069 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:06.738+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:07.070 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:06.738+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:07.070 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:06.738+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:07.070 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:06.738+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:07.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:07.094+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:07.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:07.094+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:07.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:07.094+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:07.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:07.080+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:07.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:07.080+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:07.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:07.080+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:07.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:07.080+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:08.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:07.746+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:08.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:07.746+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:08.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:07.746+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:08.081 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:07.746+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:08.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:08.092+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:08.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:08.092+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:08.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:08.092+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:08.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:08.092+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:08.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:08.116+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:08.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:08.116+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:08.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:08.116+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:08.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:08.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:08.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:08.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:08.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:08.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:08.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:08.715+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:09.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:09.164+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:09.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:09.164+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:09.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:09.164+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:09.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:09.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:09.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:09.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:09.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:09.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:09.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:09.127+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:09.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:09.698+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:09.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:09.698+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:09.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:09.698+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:09.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:09.698+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:10.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:10.104+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:10.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:10.104+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:10.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:10.104+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:10.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:10.104+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:10.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:10.206+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:10.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:10.206+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:10.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:10.206+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:10.651+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:10.651+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:10.651+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:10.651+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:11.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:11.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:11.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:11.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:11.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:11.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:11.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:11.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:11.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:11.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:11.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:11.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:11.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:11.059+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:11.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:11.888+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1164 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:11.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:11.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:11.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:11.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:11.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:11.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:11.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:11.694+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:12.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:12.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:12.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:12.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:12.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:12.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:12.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:12.014+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:12.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:12.014+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:12.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:12.014+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:12.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:12.014+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:12.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:12.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:12.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:12.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:12.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:12.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:12.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:12.651+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:13.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:13.127+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:13.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:13.127+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:13.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:13.127+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:13.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:13.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:13.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:13.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:13.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:13.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:13.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:13.032+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:13.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:13.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:13.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:13.694+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:14.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:14.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:14.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:14.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:14.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:14.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:14.006+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:14.006+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:14.006+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:14.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:14.007+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:14.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:14.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:14.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:14.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:14.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:14.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:14.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:14.711+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:15.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:15.137+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:15.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:15.137+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:15.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:15.137+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:15.022+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:15.022+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:15.022+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:15.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:15.022+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:15.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:15.716+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:15.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:15.716+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:15.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:15.716+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:15.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:15.716+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:16.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:16.133+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:16.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:16.133+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:16.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:16.133+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:16.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:16.056+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:16.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:16.056+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:16.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:16.056+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:16.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:16.056+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:16.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:16.889+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1221 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:16.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:16.705+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:16.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:16.705+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:16.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:16.705+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:16.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:16.705+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:17.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:17.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:17.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:17.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:17.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:17.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:17.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:17.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:17.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:17.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:17.016+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:18.003 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:17.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:18.003 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:17.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:18.003 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:17.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:18.004 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:17.724+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:18.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:18.201+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:18.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:18.201+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:18.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:18.201+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:18.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.015+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:19.146 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:18.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:19.146 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:18.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:19.147 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:18.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:19.147 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:18.732+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:19.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.976+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:19.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.976+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:19.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.976+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:19.148 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:18.977+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:19.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:19.158+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:19.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:19.158+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:19.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:19.158+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:20.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:19.763+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:20.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:19.763+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:20.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:19.763+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:20.151 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:19.763+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:20.151 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:19.938+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:20.151 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:19.938+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:20.151 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:19.938+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:20.151 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:19.938+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:20.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:20.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:20.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:20.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:20.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:20.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:20.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:20.771+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:20.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:20.771+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:20.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:20.771+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:20.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:20.771+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:20.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:20.929+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:20.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:20.929+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:20.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:20.929+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:20.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:20.929+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:21.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:21.161+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:21.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:21.161+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:21.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:21.161+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:22.178 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:21.889+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1280 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:22.178 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:21.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:21.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:21.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:21.774+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:21.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:21.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:21.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:22.179 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:21.896+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:22.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:22.190+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:22.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:22.190+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:22.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:22.190+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:22.807+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:22.807+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:22.807+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:22.807+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:23.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:22.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:23.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:22.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:23.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:22.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:23.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:22.909+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:23.609 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:23.227+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:23.609 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:23.227+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:23.610 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:23.227+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:24.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:23.810+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:24.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:23.810+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:24.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:23.810+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:24.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:23.810+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:24.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:23.906+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:24.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:23.906+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:24.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:23.906+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:24.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:23.906+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:24.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:24.253+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:24.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:24.253+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:24.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:24.253+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:24.938+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:24.938+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:24.938+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:24.938+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:25.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:24.779+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:25.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:24.779+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:25.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:24.779+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:25.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:24.779+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:25.245+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:25.245+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:25.245+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:26.186 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:25.959+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:26.186 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:25.959+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:26.186 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:25.959+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:26.186 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:25.959+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:26.186 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:25.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:26.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:25.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:26.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:25.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:26.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:25.799+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:26.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:26.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:26.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:26.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:26.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:26.197+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:27.162 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:26.890+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1337 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:27.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:26.830+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:27.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:26.830+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:27.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:26.830+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:27.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:26.830+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:27.163 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:26.917+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:27.163 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:26.917+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:27.164 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:26.917+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:27.164 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:26.917+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:27.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:27.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:27.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:27.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:27.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:27.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:28.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:27.916+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:28.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:27.916+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:28.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:27.916+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:28.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:27.916+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:28.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:27.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:27.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:27.798+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:27.798+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:28.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:28.217+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:28.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:28.217+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:28.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:28.217+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:29.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:28.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:29.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:28.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:29.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:28.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:29.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:28.874+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:28.829+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:28.829+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:28.829+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:29.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:28.829+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:29.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:29.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:29.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:29.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:29.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:29.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:29.862+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:29.862+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:29.862+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:29.862+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:30.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:29.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:30.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:29.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:30.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:29.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:30.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:29.874+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:30.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:30.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:30.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:31.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:30.890+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:31.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:30.890+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:30.890+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:30.890+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:30.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:30.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:30.848+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:30.848+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:31.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:31.216+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:31.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:31.216+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:31.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:31.216+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:32.163 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:31.890+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1396 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:32.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:31.876+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:32.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:31.876+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:32.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:31.876+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:32.164 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:31.876+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:32.164 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:31.905+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:32.164 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:31.905+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:32.164 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:31.905+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:32.164 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:31.905+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:32.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:32.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:32.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:32.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:32.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:32.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:33.133 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:32.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:32.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:32.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:32.914+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:32.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:32.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:32.849+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:33.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:32.849+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:33.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:33.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:33.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:33.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:33.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:33.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:34.166 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:33.952+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:34.166 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:33.952+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:34.167 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:33.952+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:34.167 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:33.952+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:34.167 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:33.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:34.167 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:33.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:34.167 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:33.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:34.167 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:33.820+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:34.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:34.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:34.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:34.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:34.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:34.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:34.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:34.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:34.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:34.989+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:35.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:34.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:35.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:34.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:35.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:34.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:35.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:34.796+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:35.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:35.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:35.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:35.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:35.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:35.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:35.990+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:35.990+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:35.990+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:35.990+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:35.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:35.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:36.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:35.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:36.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:35.748+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:36.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:36.240+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:36.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:36.240+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:36.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:36.240+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:37.029 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:36.890+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1453 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:37.029 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:36.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:37.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:36.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:37.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:36.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:37.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:36.765+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:37.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:37.235+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:37.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:37.235+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:37.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:37.235+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:37.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:37.040+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:37.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:37.040+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:37.040+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:37.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:37.040+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:37.803+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:37.803+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:37.803+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:37.803+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:38.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:38.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:38.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:38.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:38.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:38.032+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:38.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:38.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:38.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:38.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:38.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:38.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:39.015 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:38.772+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:39.015 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:38.772+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:39.015 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:38.772+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:39.015 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:38.772+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:39.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:39.239+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:39.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:39.239+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:39.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:39.239+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:39.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:39.460 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.026+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.980+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.980+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.980+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:39.980+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:40.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:39.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:40.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:39.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:40.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:39.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:40.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:39.790+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:40.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:40.288+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:40.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:40.288+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:40.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:40.288+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:40.995+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:40.995+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:40.995+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:40.995+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:41.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:40.762+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:41.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:40.762+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:41.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:40.762+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:41.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:40.762+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:41.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:41.328+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:41.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:41.328+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:41.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:41.328+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:41.999 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:41.891+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1512 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:42.000 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:41.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:42.000 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:41.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:42.000 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:41.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:42.000 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:41.724+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:42.359 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.011+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:42.359 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.011+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:42.359 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.011+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:42.359 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.011+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:42.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:42.370+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:42.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:42.370+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:42.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:42.370+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:42.978 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:42.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:42.978 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:42.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:42.978 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:42.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:42.979 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:42.725+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:43.407 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:43.407 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:43.407 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:43.407 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:42.989+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:43.669 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:43.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:43.670 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:43.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:43.670 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:43.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:43.950 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:43.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:43.950 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:43.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:43.950 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:43.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:43.950 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:43.680+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:44.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:43.961+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:43.961+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:43.961+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:43.961+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:44.665 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:44.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:44.665 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:44.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:44.665 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:44.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:44.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:44.675+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:44.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:44.675+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:44.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:44.675+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:44.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:44.676+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:45.432 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.007+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:45.432 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.007+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:45.432 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.007+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:45.432 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.007+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:45.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:45.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:45.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:45.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:45.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:45.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:45.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:45.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:45.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:45.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:45.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:45.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:45.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:45.646+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:46.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:45.998+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:46.881 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:46.475+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:46.881 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:46.475+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:46.882 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:46.475+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:46.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:46.666+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:46.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:46.666+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:46.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:46.666+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:46.882 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:46.666+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:47.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:46.892+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1569 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:47.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:47.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:47.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:47.026+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:47.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:47.462+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:47.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:47.462+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:47.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:47.462+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:47.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:47.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:47.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:47.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:47.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:47.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:47.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:47.680+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:48.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:48.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:48.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:48.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:48.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:48.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:48.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:48.021+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:48.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:48.495+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:48.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:48.495+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:48.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:48.495+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:48.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:48.730+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:48.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:48.730+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:48.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:48.730+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:48.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:48.730+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:49.437 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.039+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:49.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.039+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:49.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.039+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:49.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.039+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:49.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:49.449+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:49.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:49.449+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:49.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:49.449+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:49.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:49.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:49.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:49.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:49.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:49.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:49.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:49.694+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:50.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.993+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:50.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.993+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:50.402 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.993+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:50.402 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:49.993+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:50.689 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:50.413+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:50.690 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:50.413+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:50.690 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:50.413+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:50.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:50.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:50.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:50.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:50.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:50.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:50.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:50.701+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:51.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:50.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:51.431 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:50.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:51.431 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:50.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:51.432 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:50.982+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:51.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:51.419+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:51.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:51.419+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:51.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:51.419+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:52.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:51.893+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1628 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:51.948+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:51.948+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:51.948+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:51.948+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:51.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:51.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:51.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:51.733+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:52.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:52.452+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:52.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:52.452+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:52.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:52.452+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:53.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:52.977+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:53.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:52.977+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:53.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:52.977+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:53.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:52.977+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:53.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:52.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:53.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:52.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:53.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:52.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:53.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:52.733+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:53.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:53.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:53.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:53.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:53.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:53.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:54.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:54.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:54.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:54.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:54.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:54.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:54.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:54.000+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:54.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:53.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:54.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:53.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:54.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:53.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:54.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:53.755+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:54.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:54.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:54.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:54.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:54.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:54.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:54.997 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:54.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:54.997 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:54.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:54.997 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:54.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:54.997 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:54.740+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:55.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.008+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:55.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.008+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:55.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.008+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:55.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.008+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:55.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:55.489+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:55.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:55.489+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:55.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:55.489+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:55.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:55.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:55.692+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:55.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:55.692+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:56.454 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.990+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:56.454 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.990+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:56.454 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.990+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:56.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:55.990+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:56.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:56.466+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:56.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:56.466+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:56.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:56.466+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:56.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:56.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:56.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:56.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:56.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:56.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:56.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:56.701+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:57.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:05:56.894+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1685 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:05:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:56.996+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:56.996+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:56.996+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:56.996+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:57.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:57.484+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:57.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:57.484+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:57.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:57.484+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:57.709+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:57.709+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:57.709+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:57.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:57.710+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:57.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:58.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:57.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:58.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:57.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:58.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:57.997+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:58.951 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:58.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:58.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:58.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:58.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:58.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:05:58.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:58.724+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:58.952 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:58.488+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:58.952 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:58.488+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:58.952 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:58.488+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:59.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:58.963+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:59.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:58.963+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:59.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:58.963+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:05:59.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:58.963+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:05:59.754 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:59.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:59.754 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:59.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:05:59.754 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:05:59.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:59.933+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:59.933+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:59.933+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:05:59.933+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:00.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:59.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:00.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:59.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:00.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:59.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:00.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:05:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:05:59.766+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:00.762 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:00.502+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:00.763 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:00.502+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:00.763 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:00.502+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:01.070 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:00.925+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:01.070 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:00.925+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:01.070 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:00.925+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:01.070 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:00.925+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:01.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:00.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:01.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:00.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:01.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:00.774+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:01.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:00.774+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:01.743 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:01.484+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:01.743 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:01.484+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:01.743 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:01.484+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:02.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:01.895+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1744 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:01.912+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:01.912+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:01.912+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:01.912+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:02.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:01.754+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:02.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:01.754+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:02.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:01.754+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:02.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:01.754+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:02.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:02.453+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:02.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:02.453+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:02.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:02.453+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:02.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:02.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:02.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:02.879+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:03.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:02.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:03.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:02.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:03.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:02.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:03.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:02.796+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:03.761 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:03.485+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:03.761 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:03.485+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:03.762 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:03.485+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:03.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:03.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:03.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:03.874+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:04.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:03.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:04.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:03.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:04.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:03.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:04.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:03.773+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:04.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:04.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:04.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:04.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:04.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:04.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:04.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:04.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:04.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:04.914+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:05.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:04.749+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:05.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:04.749+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:05.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:04.749+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:05.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:04.749+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:05.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:05.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:05.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:05.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:05.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:05.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:05.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:05.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:05.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:05.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:05.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:05.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:05.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:05.703+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:05.935+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:05.935+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:05.935+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:05.935+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:06.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:06.434+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:06.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:06.434+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:06.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:06.435+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:06.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:06.671+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:06.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:06.671+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:06.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:06.671+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:06.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:06.671+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:07.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:06.895+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1801 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:06.959+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:06.959+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:06.959+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:06.959+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:07.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:07.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:07.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:07.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:07.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:07.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:07.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:07.629+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:07.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:07.629+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:07.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:07.629+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:07.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:07.629+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:08.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:07.920+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:07.920+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:07.920+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:07.920+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:08.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:08.444+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:08.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:08.444+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:08.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:08.444+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:08.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:08.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:08.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:08.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:08.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:08.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:08.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:08.676+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:09.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:08.881+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:09.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:08.881+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:09.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:08.881+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:09.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:08.881+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:09.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:09.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:09.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:09.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:09.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:09.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:09.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:09.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:09.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:09.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:09.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:09.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:09.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:09.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:09.929+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:09.929+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:09.929+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:09.929+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:10.660 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:10.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:10.660 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:10.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:10.661 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:10.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:10.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:10.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:10.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:10.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:10.891+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:10.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:10.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:10.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:10.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:10.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:10.672+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:11.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:11.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:11.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:11.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:11.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:11.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:11.884 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:11.632+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:11.885 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:11.632+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:11.885 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:11.632+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:11.885 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:11.632+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:11.885 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:11.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:11.885 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:11.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:11.885 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:11.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:11.886 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:11.857+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:12.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:11.896+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1860 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:12.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:12.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:12.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:12.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:12.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:12.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:12.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:12.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:12.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:12.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:12.610+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:13.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:12.815+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:13.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:12.815+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:13.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:12.815+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:13.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:12.815+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:13.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:13.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:13.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:13.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:13.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:13.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:13.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:13.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:13.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:13.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:13.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:13.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:13.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:13.570+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:14.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:13.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:14.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:13.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:14.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:13.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:14.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:13.841+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:14.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:14.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:14.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:14.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:14.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:14.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:14.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:14.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:14.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:14.566+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:15.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:14.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:15.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:14.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:15.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:14.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:15.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:14.859+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:15.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:15.409+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:15.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:15.409+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:15.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:15.409+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:15.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:15.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:15.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:15.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:15.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:15.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:15.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:15.590+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:16.089 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:15.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:16.089 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:15.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:16.089 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:15.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:16.090 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:15.813+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:16.613 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:16.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:16.613 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:16.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:16.613 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:16.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:16.885 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:16.625+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:16.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:16.625+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:16.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:16.625+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:16.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:16.625+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:16.886 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:16.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:16.886 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:16.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:16.886 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:16.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:16.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:16.817+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:17.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:16.897+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1917 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:17.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:17.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:17.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:17.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:17.604+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:17.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:17.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:17.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:17.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:17.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:17.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:18.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:17.802+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:18.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:17.802+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:18.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:17.802+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:18.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:17.802+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:18.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:18.359+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:18.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:18.359+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:18.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:18.359+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:18.578+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:18.578+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:18.578+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:18.578+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:19.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:18.827+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:19.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:18.827+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:19.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:18.827+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:19.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:18.827+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:19.592 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:19.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:19.592 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:19.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:19.592 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:19.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:19.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:19.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:19.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:19.782+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:19.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:19.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:19.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:19.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:19.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:19.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:19.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:19.604+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:20.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:20.365+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:20.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:20.365+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:20.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:20.365+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:20.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:20.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:20.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:20.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:20.614+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:21.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:20.802+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:21.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:20.802+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:21.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:20.802+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:21.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:20.802+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:21.635 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:21.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:21.635 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:21.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:21.635 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:21.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:21.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:21.647+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:21.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:21.647+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:21.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:21.647+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:21.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:21.647+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:21.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:21.798+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:21.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:21.798+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:21.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:21.798+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:21.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:21.799+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:22.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:21.896+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 1976 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:22.619 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:22.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:22.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:22.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:22.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:22.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:22.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:22.631+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:22.631+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:22.631+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:22.631+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:22.796+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:22.796+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:22.796+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:22.796+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:23.405+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:23.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:23.405+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:23.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:23.405+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:23.591+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:23.591+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:23.591+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:23.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:23.591+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:23.747+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:23.747+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:24.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:23.747+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:24.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:23.748+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:24.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:24.436+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:24.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:24.436+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:24.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:24.436+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:24.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:24.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:24.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:24.566+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:24.724+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:24.724+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:24.724+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:24.724+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:25.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:25.468+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:25.468+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:25.468+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:25.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:25.534+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:25.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:25.534+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:25.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:25.534+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:25.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:25.534+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:25.727+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:25.727+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:26.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:25.727+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:26.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:25.727+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:26.674 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:26.430+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:26.674 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:26.430+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:26.674 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:26.430+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:26.675 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:26.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:26.675 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:26.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:26.675 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:26.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:26.675 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:26.526+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:26.958 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:26.897+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2033 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:26.685+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:26.685+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:26.685+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:26.685+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:27.703 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:27.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:27.703 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:27.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:27.703 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:27.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:27.704 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:27.504+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:27.704 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:27.504+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:27.704 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:27.504+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:27.704 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:27.504+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:27.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:27.714+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:27.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:27.714+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:27.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:27.714+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:27.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:27.714+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:28.679 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:28.360+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:28.679 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:28.360+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:28.679 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:28.360+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:28.679 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:28.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:28.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:28.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:28.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:28.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:28.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:28.485+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:28.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:28.690+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:28.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:28.690+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:28.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:28.690+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:28.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:28.690+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:29.685 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:29.375+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:29.685 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:29.375+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:29.685 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:29.375+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:29.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:29.502+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:29.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:29.502+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:29.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:29.502+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:29.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:29.502+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:29.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:29.696+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:29.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:29.696+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:29.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:29.696+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:29.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:29.696+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:30.417+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:30.417+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:30.417+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:30.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:30.665+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:30.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:30.665+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:30.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:30.665+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:30.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:30.665+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:30.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:30.512+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:30.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:30.512+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:30.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:30.512+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:30.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:30.512+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:31.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:31.389+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:31.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:31.389+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:31.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:31.389+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:31.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:31.622+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:31.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:31.622+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:31.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:31.622+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:31.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:31.622+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:31.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:31.560+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:31.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:31.560+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:31.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:31.560+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:31.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:31.560+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:32.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:31.898+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2093 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:32.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:32.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:32.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:32.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:32.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:32.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:32.602+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:32.602+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:32.602+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:32.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:32.602+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:32.536+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:32.536+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:32.536+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:32.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:32.536+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:33.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:33.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:33.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:33.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:33.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:33.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:33.617+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:33.617+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:33.617+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:33.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:33.617+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:33.489+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:33.489+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:33.489+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:33.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:33.489+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:34.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:34.439+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:34.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:34.439+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:34.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:34.439+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:34.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:34.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:34.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:34.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:34.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:34.569+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:34.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:34.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:34.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:34.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:34.485+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:35.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:35.420+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:35.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:35.420+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:35.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:35.420+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:35.610+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:35.610+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:35.610+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:35.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:35.610+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:35.473+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:35.473+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:35.473+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:35.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:35.473+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:36.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:36.451+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:36.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:36.451+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:36.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:36.451+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:36.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:36.431+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:36.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:36.431+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:36.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:36.431+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:36.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:36.431+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:36.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:36.599+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:36.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:36.599+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:36.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:36.599+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:36.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:36.599+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:37.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:36.899+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2150 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:37.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:37.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:37.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:37.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:37.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:37.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:37.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:37.462+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:37.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:37.462+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:37.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:37.462+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:37.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:37.462+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:37.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:37.588+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:37.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:37.588+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:37.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:37.588+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:37.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:37.588+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:38.530+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:38.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:38.530+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:38.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:38.530+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:38.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:38.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:38.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:38.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:38.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:38.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:38.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:38.568+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:38.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:38.416+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:38.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:38.416+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:38.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:38.416+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:38.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:38.416+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:39.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:39.518+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:39.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:39.518+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:39.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:39.518+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:39.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:39.384+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:39.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:39.384+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:39.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:39.384+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:39.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:39.384+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:39.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:39.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:39.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:39.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:39.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:39.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:39.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:39.595+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:40.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:40.544+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:40.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:40.544+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:40.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:40.544+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:40.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:40.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:40.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:40.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:40.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:40.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:40.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:40.379+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:40.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:40.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:40.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:40.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:40.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:40.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:40.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:40.600+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:41.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:41.512+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:41.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:41.512+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:41.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:41.512+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:41.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:41.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:41.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:41.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:41.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:41.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:41.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:41.379+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:41.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:41.552+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:41.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:41.552+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:41.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:41.552+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:41.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:41.552+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:42.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:41.900+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2210 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:42.536 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:42.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:42.536 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:42.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:42.536 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:42.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:42.536 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:42.379+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:42.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:42.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:42.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:42.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:42.551+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:42.551+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:42.551+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:42.551+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:43.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:43.577+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:43.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:43.577+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:43.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:43.577+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:43.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:43.540+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:43.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:43.540+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:43.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:43.540+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:43.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:43.540+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:43.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:43.351+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:43.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:43.351+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:43.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:43.351+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:43.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:43.351+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:44.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:44.548+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:44.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:44.548+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:44.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:44.548+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:44.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:44.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:44.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:44.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:44.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:44.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:44.710 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:44.536+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:44.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:44.312+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:44.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:44.312+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:44.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:44.312+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:44.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:44.312+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:45.568 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:45.282+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:45.568 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:45.282+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:45.568 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:45.282+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:45.568 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:45.282+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:45.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:45.587+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:45.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:45.588+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:45.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:45.588+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:45.580+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:45.580+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:45.580+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:45.580+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:46.513 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:46.241+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:46.513 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:46.241+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:46.513 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:46.241+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:46.513 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:46.241+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:46.890 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:46.636+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:46.890 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:46.636+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:46.890 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:46.636+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:46.890 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:46.542+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:46.891 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:46.542+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:46.891 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:46.542+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:46.891 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:46.542+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:47.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:46.901+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2268 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:47.545 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:47.231+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:47.545 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:47.231+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:47.545 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:47.231+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:47.545 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:47.231+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:47.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:47.557+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:47.557+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:47.557+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:47.557+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:47.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:47.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:47.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:47.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:47.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:47.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:48.550 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:48.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:48.550 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:48.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:48.550 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:48.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:48.550 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:48.255+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:48.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:48.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:48.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:48.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:48.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:48.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:48.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:48.562+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:48.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:48.683+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:48.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:48.683+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:48.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:48.683+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:49.218+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:49.218+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:49.218+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:49.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:49.218+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:49.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:49.524+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:49.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:49.524+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:49.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:49.524+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:49.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:49.524+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:49.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:49.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:49.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:49.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:49.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:49.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:50.520 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:50.262+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:50.520 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:50.262+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:50.521 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:50.262+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:50.521 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:50.262+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:50.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:50.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:50.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:50.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:50.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:50.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:50.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:50.532+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:50.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:50.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:50.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:50.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:50.960 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:50.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:51.214+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:51.214+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:51.214+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:51.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:51.215+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:51.891 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:51.662+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:51.891 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:51.662+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:51.891 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:51.662+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:51.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:51.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:51.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:51.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:51.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:51.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:51.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:51.505+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:52.181 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:51.902+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2327 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:52.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:52.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:52.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:52.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:52.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:52.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:52.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:52.193+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:52.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:52.689+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:52.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:52.689+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:52.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:52.689+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:52.962 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:52.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:52.962 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:52.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:52.962 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:52.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:52.962 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:52.490+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:53.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:53.160+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:53.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:53.160+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:53.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:53.160+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:53.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:53.160+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:53.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:53.669+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:53.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:53.669+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:53.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:53.669+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:53.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:53.526+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:53.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:53.526+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:53.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:53.526+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:53.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:53.526+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:54.186+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:54.186+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:54.186+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:54.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:54.186+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:54.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:54.630+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:54.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:54.630+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:54.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:54.630+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:54.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:54.510+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:54.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:54.510+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:54.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:54.510+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:54.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:54.510+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:55.452 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:55.167+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:55.452 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:55.167+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:55.452 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:55.167+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:55.452 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:55.167+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:55.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:55.608+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:55.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:55.608+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:55.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:55.608+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:55.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:55.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:55.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:55.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:55.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:55.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:55.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:55.464+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:56.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:56.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:56.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:56.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:56.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:56.163+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:56.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:56.163+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:56.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:56.647+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:56.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:56.647+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:56.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:56.647+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:56.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:56.423+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:56.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:56.423+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:56.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:56.423+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:56.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:56.423+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:57.161 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:06:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:06:56.903+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2384 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:06:57.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:57.173+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:57.173+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:57.173+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:57.173+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:57.423+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:57.423+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:57.423+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:57.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:57.423+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:57.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:57.663+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:57.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:57.663+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:57.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:57.663+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:58.379+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:58.379+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:58.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:58.379+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:58.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:58.379+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:58.202+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:58.202+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:58.202+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:58.202+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:58.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:58.667+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:58.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:58.667+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:58.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:58.667+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:59.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:59.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:59.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:59.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:59.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:59.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:06:59.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:06:59.351+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:59.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:59.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:59.193+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:06:59.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:06:59.193+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:06:59.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:59.654+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:59.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:59.654+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:06:59.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:06:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:06:59.654+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:00.388+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:00.388+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:00.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:00.388+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:00.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:00.388+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:00.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:00.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:00.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:00.191+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:00.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:00.699+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:00.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:00.699+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:00.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:00.699+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:01.225+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:01.225+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:01.225+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:01.225+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:01.660 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:01.438+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:01.660 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:01.438+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:01.660 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:01.438+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:01.660 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:01.438+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:01.958 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:01.904+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2443 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:01.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:01.718+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:01.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:01.718+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:01.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:01.718+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:02.662 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:02.243+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:02.662 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:02.243+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:02.662 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:02.243+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:02.663 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:02.243+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:02.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:02.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:02.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:02.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:02.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:02.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:02.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:02.485+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:02.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:02.674+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:02.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:02.674+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:02.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:02.674+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:03.500 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:03.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:03.500 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:03.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:03.500 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:03.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:03.500 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:03.236+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:03.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:03.511+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:03.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:03.511+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:03.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:03.512+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:03.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:03.512+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:03.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:03.688+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:03.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:03.688+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:03.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:03.688+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:04.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:04.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:04.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:04.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:04.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:04.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:04.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:04.492+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:04.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:04.279+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:04.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:04.279+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:04.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:04.279+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:04.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:04.279+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:04.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:04.653+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:04.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:04.653+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:04.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:04.653+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:05.516 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:05.240+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:05.516 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:05.240+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:05.517 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:05.240+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:05.517 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:05.240+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:05.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:05.528+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:05.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:05.528+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:05.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:05.528+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:05.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:05.528+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:05.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:05.674+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:05.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:05.674+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:05.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:05.674+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:06.495 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:06.252+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:06.495 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:06.252+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:06.495 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:06.252+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:06.496 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:06.252+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:06.892 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:06.696+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:06.893 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:06.696+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:06.893 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:06.696+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:06.893 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:06.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:06.893 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:06.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:06.893 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:06.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:06.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:06.507+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:07.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:06.904+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2500 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:07.537 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:07.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:07.538 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:07.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:07.538 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:07.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:07.538 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:07.291+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:07.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:07.651+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:07.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:07.651+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:07.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:07.651+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:07.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:07.550+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:07.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:07.550+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:07.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:07.550+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:07.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:07.550+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:08.581 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:08.336+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:08.582 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:08.336+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:08.582 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:08.336+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:08.582 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:08.336+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:08.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:08.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:08.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:08.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:08.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:08.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:08.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:08.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:08.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:08.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:08.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:08.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:08.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:08.594+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:09.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:09.341+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:09.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:09.341+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:09.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:09.341+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:09.604 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:09.341+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:09.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:09.615+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:09.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:09.615+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:09.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:09.615+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:09.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:09.633+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:09.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:09.633+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:09.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:09.633+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:09.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:09.633+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:10.628 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:10.376+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:10.628 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:10.376+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:10.628 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:10.376+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:10.628 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:10.376+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:10.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:10.640+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:10.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:10.640+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:10.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:10.640+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:10.663+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:10.663+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:10.663+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:10.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:10.663+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:11.614 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:11.334+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:11.614 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:11.334+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:11.614 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:11.334+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:11.615 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:11.334+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:11.893 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:11.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:11.894 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:11.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:11.894 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:11.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:11.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:11.626+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:11.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:11.626+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:11.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:11.626+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:11.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:11.626+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:12.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:11.905+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2559 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:12.616 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:12.310+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:12.616 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:12.310+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:12.616 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:12.310+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:12.616 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:12.310+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:12.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:12.656+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:12.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:12.656+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:12.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:12.656+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:12.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:12.628+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:12.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:12.628+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:12.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:12.628+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:12.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:12.628+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:13.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:13.274+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:13.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:13.274+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:13.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:13.274+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:13.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:13.274+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:13.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:13.639+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:13.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:13.639+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:13.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:13.639+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:13.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:13.616+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:13.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:13.616+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:13.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:13.616+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:13.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:13.616+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:14.599 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:14.320+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:14.599 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:14.320+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:14.599 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:14.320+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:14.599 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:14.321+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:14.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:14.678+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:14.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:14.678+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:14.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:14.678+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:14.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:14.610+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:14.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:14.610+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:14.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:14.610+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:14.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:14.610+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:15.552 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:15.295+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:15.552 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:15.295+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:15.552 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:15.295+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:15.552 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:15.295+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:15.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:15.670+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:15.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:15.670+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:15.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:15.670+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:15.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:15.563+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:15.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:15.563+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:15.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:15.563+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:15.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:15.563+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:16.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:16.249+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:16.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:16.249+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:16.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:16.249+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:16.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:16.249+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:16.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:16.609+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:16.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:16.609+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:16.894 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:16.609+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:16.895 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:16.609+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:16.895 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:16.660+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:16.895 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:16.660+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:16.895 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:16.660+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:17.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:16.905+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2621 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:17.609 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:17.276+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:17.609 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:17.276+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:17.609 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:17.276+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:17.609 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:17.276+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:17.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:17.620+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:17.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:17.620+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:17.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:17.620+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:17.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:17.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:17.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:17.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:17.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:17.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:17.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:17.642+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:18.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:18.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:18.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:18.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:18.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:18.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:18.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:18.233+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:18.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:18.594+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:18.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:18.594+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:18.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:18.594+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:18.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:18.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:18.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:18.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:18.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:18.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:18.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:18.661+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:19.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:19.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:19.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:19.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:19.195+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:19.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:19.569+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:19.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:19.569+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:19.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:19.569+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:19.690+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:19.690+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:19.690+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:19.690+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:20.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:20.216+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:20.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:20.216+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:20.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:20.216+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:20.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:20.216+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:20.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:20.524+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:20.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:20.524+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:20.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:20.524+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:20.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:20.697+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:20.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:20.697+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:20.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:20.697+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:20.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:20.697+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:21.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:21.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:21.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:21.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:21.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:21.213+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:21.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:21.213+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:21.855 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:21.490+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:21.856 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:21.490+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:21.856 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:21.490+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:21.856 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:21.738+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:21.856 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:21.738+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:21.856 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:21.738+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:21.856 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:21.738+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:22.176 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:21.905+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2680 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:22.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:22.188+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:22.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:22.188+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:22.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:22.188+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:22.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:22.188+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:22.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:22.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:22.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:22.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:22.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:22.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:23.181 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:22.766+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:23.182 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:22.766+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:23.182 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:22.766+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:23.182 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:22.766+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:23.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:23.192+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:23.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:23.192+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:23.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:23.192+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:23.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:23.192+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:23.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:23.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:23.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:24.179 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:23.784+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:24.180 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:23.784+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:24.180 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:23.784+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:24.180 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:23.784+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:24.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:24.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:24.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:24.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:24.191+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:24.798 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:24.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:24.798 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:24.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:24.798 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:24.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:25.155 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:24.810+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:25.155 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:24.810+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:25.155 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:24.810+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:25.155 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:24.810+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:25.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:25.167+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:25.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:25.167+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:25.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:25.167+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:25.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:25.167+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:25.843 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:25.475+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:25.843 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:25.475+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:25.844 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:25.475+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:26.184 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:25.855+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:26.185 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:25.855+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:26.185 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:25.855+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:26.185 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:25.855+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:26.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:26.195+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:26.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:26.196+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:26.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:26.196+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:26.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:26.196+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:26.886 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:26.478+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:26.887 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:26.478+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:26.887 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:26.479+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:27.169 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:26.906+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2740 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:27.169 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:26.898+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:27.170 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:26.898+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:27.170 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:26.898+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:27.170 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:26.898+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:27.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:27.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:27.451 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:27.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:27.451 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:27.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:27.451 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:27.181+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:27.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:27.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:27.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:27.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:27.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:27.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:28.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:27.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:27.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:27.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:27.909+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:28.148+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:28.148+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:28.148+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:28.149+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:28.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:28.435+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:28.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:28.435+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:28.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:28.435+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:29.160 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:28.864+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:29.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:28.864+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:29.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:28.864+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:29.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:28.864+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:29.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:29.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:29.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:29.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:29.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:29.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:29.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:29.171+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:29.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:29.171+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:29.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:29.171+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:29.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:29.171+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:30.112 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:29.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:30.113 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:29.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:30.113 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:29.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:30.113 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:29.857+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:30.418 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:30.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:30.418 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:30.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:30.418 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:30.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:30.418 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:30.123+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:30.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:30.430+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:30.430+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:30.430+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:31.123 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:30.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:31.124 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:30.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:31.124 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:30.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:31.124 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:30.817+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:31.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:31.135+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:31.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:31.135+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:31.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:31.135+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:31.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:31.135+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:31.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:31.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:31.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:31.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:31.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:31.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:32.085 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:31.907+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2800 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:32.085 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:31.786+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:32.085 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:31.786+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:32.085 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:31.786+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:32.086 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:31.786+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:32.366 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:32.097+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:32.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:32.097+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:32.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:32.097+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:32.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:32.097+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:32.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:32.378+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:32.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:32.378+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:32.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:32.378+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:33.086 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:32.836+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:33.086 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:32.836+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:33.086 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:32.836+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:33.087 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:32.836+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:33.381 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:33.098+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:33.098+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:33.098+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:33.098+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:33.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:33.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:33.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:33.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:33.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:33.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:34.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:33.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:34.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:33.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:34.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:33.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:34.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:33.859+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:34.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:34.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:34.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:34.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:34.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:34.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:34.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:34.059+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:34.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:34.390+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:34.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:34.390+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:34.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:34.390+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:35.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:34.858+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:34.858+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:34.858+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:35.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:34.858+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:35.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:35.021+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:35.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:35.021+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:35.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:35.021+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:35.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:35.021+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:35.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:35.419+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:35.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:35.419+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:35.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:35.419+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:36.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:35.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:35.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:35.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:35.863+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:36.066+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:36.066+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:36.066+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:36.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:36.066+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:36.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:36.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:36.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:36.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:36.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:36.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:37.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:36.909+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2858 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:36.880+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:36.880+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:36.880+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:36.880+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:37.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:37.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:37.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:37.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:37.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:37.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:37.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:37.100+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:37.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:37.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:37.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:37.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:37.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:37.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:38.103 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:37.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:38.103 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:37.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:38.103 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:37.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:38.103 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:37.841+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:38.357+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:38.357+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:38.357+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:38.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:38.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:38.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:38.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:38.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:38.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:38.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:38.115+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:38.883+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:38.883+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:39.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:38.883+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:39.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:38.883+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:39.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:39.101+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:39.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:39.101+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:39.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:39.101+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:39.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:39.101+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:39.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:39.313+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:39.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:39.313+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:39.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:39.313+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:39.931+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:39.931+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:39.931+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:39.931+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:40.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:40.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:40.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:40.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:40.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:40.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:40.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:40.077+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:40.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:40.279+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:40.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:40.279+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:40.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:40.279+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:41.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:40.913+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:40.913+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:40.913+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:40.913+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:41.071+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:41.071+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:41.071+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:41.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:41.071+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:41.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:41.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:41.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:41.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:41.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:41.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:42.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:41.910+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2921 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:41.954+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:41.954+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:41.954+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:41.954+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:42.049+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:42.049+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:42.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:42.049+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:42.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:42.050+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:42.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:42.286+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:42.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:42.286+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:42.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:42.286+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:43.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:42.937+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:43.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:42.937+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:43.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:42.937+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:43.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:42.937+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:43.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:43.007+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:43.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:43.007+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:43.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:43.007+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:43.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:43.007+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:43.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:43.249+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:43.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:43.249+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:43.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:43.249+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:44.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:43.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:43.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:43.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:43.970+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:44.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:44.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:44.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:44.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:44.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:44.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:44.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:44.028+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:44.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:44.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:44.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:44.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:44.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:44.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:45.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:44.925+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:45.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:44.925+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:45.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:44.925+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:45.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:44.925+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:45.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:45.029+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:45.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:45.030+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:45.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:45.030+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:45.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:45.030+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:45.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:45.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:45.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:45.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:45.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:45.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:46.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:45.930+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:46.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:45.930+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:45.930+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:45.931+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:46.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:46.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:46.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:46.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:46.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:46.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:46.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:46.001+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:46.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:46.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:46.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:46.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:46.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:46.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:47.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:46.910+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 2984 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:46.903+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:46.903+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:46.903+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:46.903+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:47.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:47.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:47.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:47.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:47.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:47.041+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:47.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:47.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:47.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:47.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:47.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:47.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:48.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:47.884+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:48.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:47.885+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:48.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:47.885+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:48.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:47.885+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:48.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:48.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:48.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:48.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:48.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:48.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:48.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:48.042+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:48.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:48.355+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:48.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:48.355+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:48.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:48.355+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:49.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:48.837+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:49.213 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:48.837+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:49.213 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:48.837+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:49.213 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:48.838+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:49.213 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:49.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:49.213 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:49.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:49.214 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:49.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:49.214 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:49.047+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:49.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:49.365+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:49.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:49.365+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:49.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:49.365+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:50.080 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:49.806+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:50.081 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:49.806+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:50.081 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:49.806+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:50.081 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:49.806+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:50.357 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:50.093+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:50.357 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:50.093+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:50.357 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:50.093+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:50.357 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:50.093+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:50.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:50.369+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:50.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:50.369+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:50.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:50.369+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:51.087 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:50.771+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:51.087 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:50.771+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:51.088 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:50.771+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:51.088 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:50.772+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:51.345 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:51.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:51.345 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:51.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:51.345 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:51.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:51.346 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:51.100+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:51.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:51.357+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:51.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:51.357+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:51.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:51.357+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:52.149 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:51.911+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3055 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:52.149 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:52.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:52.149 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:52.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:52.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:52.059+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:52.150 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:52.059+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:52.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:51.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:52.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:51.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:52.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:51.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:52.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:51.817+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:52.396+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:52.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:52.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:52.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:53.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:53.036+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:53.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:53.036+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:53.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:53.036+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:53.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:53.036+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:53.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:52.811+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:53.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:52.811+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:53.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:52.811+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:53.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:52.811+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:53.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:53.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:53.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:53.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:53.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:53.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:54.068 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:53.776+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:54.069 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:53.776+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:54.069 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:53.776+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:54.069 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:53.776+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:54.350 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:54.080+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:54.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:54.080+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:54.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:54.080+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:54.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:54.080+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:54.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:54.362+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:54.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:54.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:54.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:54.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:55.116 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:54.822+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:55.116 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:54.822+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:55.117 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:54.822+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:55.117 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:54.822+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:55.325+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:55.325+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:55.325+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:55.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:55.129+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:55.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:55.129+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:55.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:55.129+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:55.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:55.129+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:56.143 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:55.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:56.144 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:55.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:56.144 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:55.817+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:56.144 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:55.817+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:56.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:56.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:56.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:56.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:56.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:56.156+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:56.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:56.156+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:56.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:56.156+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:56.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:56.156+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:57.098 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:07:56.912+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3138 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:07:57.099 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:56.855+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:57.099 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:56.855+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:57.099 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:56.855+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:57.099 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:56.856+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:57.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:57.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:57.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:57.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:57.111+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:57.111+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:57.111+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:57.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:57.111+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:58.116 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:57.848+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:58.116 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:57.848+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:58.116 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:57.848+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:58.117 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:57.848+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:58.324+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:58.324+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:58.324+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:58.128+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:58.128+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:58.128+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:58.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:58.128+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:59.160 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:58.868+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:59.160 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:58.868+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:59.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:58.868+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:07:59.161 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:58.868+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:07:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:59.318+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:59.318+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:07:59.318+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:07:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:59.173+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:59.173+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:59.173+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:07:59.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:07:59.173+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:00.154 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:59.889+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:00.154 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:59.889+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:00.154 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:59.889+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:00.154 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:07:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:07:59.889+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:00.301+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:00.301+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:00.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:00.301+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:00.166+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:00.166+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:00.166+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:00.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:00.166+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:01.117+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:01.117+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:01.117+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:01.117+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:01.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:00.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:01.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:00.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:01.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:00.874+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:01.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:00.874+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:01.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:01.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:01.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:01.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:01.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:01.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:02.085 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:01.913+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3197 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:02.086 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:01.844+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:02.086 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:01.844+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:02.087 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:01.844+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:02.087 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:01.844+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:02.358 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:02.098+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:02.359 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:02.098+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:02.359 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:02.098+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:02.359 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:02.098+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:02.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:02.371+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:02.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:02.371+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:02.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:02.371+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:03.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:03.093+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:03.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:03.093+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:03.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:03.093+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:03.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:03.093+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:02.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:02.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:02.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:02.863+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:03.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:03.342+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:03.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:03.342+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:03.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:03.342+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:03.880+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:03.880+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:03.880+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:03.880+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:04.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:04.085+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:04.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:04.085+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:04.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:04.085+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:04.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:04.085+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:04.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:04.373+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:04.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:04.373+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:04.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:04.373+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:05.095 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:04.848+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:05.095 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:04.848+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:05.095 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:04.848+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:05.096 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:04.848+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:05.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:05.336+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:05.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:05.336+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:05.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:05.336+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:05.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:05.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:05.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:05.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:05.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:05.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:05.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:05.107+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:06.110 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:05.825+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:06.110 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:05.825+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:06.110 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:05.825+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:06.110 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:05.825+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:06.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:06.348+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:06.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:06.348+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:06.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:06.348+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:06.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:06.121+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:06.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:06.121+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:06.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:06.121+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:06.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:06.121+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:07.157 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:06.913+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3254 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:07.157 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:06.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:07.157 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:06.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:07.158 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:06.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:07.158 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:06.846+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:07.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:07.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:07.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:07.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:07.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:07.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:07.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:07.169+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:07.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:07.169+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:07.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:07.169+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:07.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:07.169+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:08.176 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:07.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:08.176 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:07.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:08.176 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:07.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:08.176 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:07.891+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:08.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:08.337+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:08.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:08.337+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:08.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:08.337+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:08.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:08.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:08.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:08.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:08.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:08.187+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:08.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:08.187+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:09.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:08.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:09.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:08.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:09.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:08.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:09.147 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:08.879+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:09.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:09.330+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:09.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:09.330+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:09.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:09.330+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:09.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:09.158+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:09.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:09.158+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:09.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:09.158+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:09.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:09.158+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:10.191 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:09.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:10.191 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:09.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:10.191 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:09.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:10.192 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:09.909+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:10.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:10.301+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:10.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:10.301+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:10.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:10.301+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:10.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:10.203+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:10.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:10.203+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:10.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:10.203+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:10.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:10.203+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:11.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:10.912+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:11.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:10.913+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:11.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:10.913+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:11.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:10.913+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:11.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:11.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:11.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:11.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:11.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:11.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:11.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:11.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:11.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:11.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:11.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:11.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:11.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:11.217+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:12.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:11.913+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3313 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:11.886+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:11.886+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:11.886+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:12.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:11.886+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:12.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:12.300+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:12.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:12.300+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:12.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:12.300+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:12.257+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:12.257+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:12.257+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:12.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:12.257+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:13.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:12.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:13.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:12.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:13.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:12.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:13.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:12.846+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:13.528 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:13.347+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:13.528 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:13.347+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:13.529 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:13.347+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:13.529 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:13.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:13.529 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:13.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:13.529 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:13.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:13.529 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:13.237+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:14.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:13.800+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:14.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:13.800+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:14.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:13.800+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:14.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:13.800+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:14.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:14.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:14.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:14.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:14.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:14.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:14.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:14.287+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:14.287+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:14.287+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:14.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:14.287+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:15.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:14.838+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:15.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:14.838+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:15.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:14.838+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:15.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:14.838+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:15.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:15.426+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:15.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:15.426+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:15.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:15.426+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:15.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:15.278+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:15.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:15.278+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:15.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:15.278+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:15.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:15.278+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:16.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:15.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:16.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:15.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:16.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:15.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:16.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:15.804+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:16.600 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:16.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:16.601 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:16.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:16.601 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:16.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:16.601 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:16.266+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:16.601 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:16.266+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:16.601 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:16.266+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:16.602 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:16.266+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:17.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:16.914+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3369 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:17.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:16.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:17.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:16.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:17.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:16.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:17.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:16.813+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:17.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:17.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:17.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:17.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:17.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:17.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:17.225+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:17.225+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:17.225+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:17.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:17.226+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:18.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:17.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:18.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:17.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:18.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:17.841+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:18.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:17.841+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:18.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:18.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:18.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:18.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:18.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:18.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:18.238+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:18.238+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:18.238+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:18.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:18.238+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:19.192 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:18.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:19.193 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:18.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:19.193 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:18.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:19.193 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:18.804+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:19.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:19.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:19.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:19.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:19.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:19.432+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:19.204+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:19.204+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:19.204+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:19.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:19.204+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:19.805+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:19.805+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:19.805+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:19.805+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:20.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:20.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:20.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:20.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:20.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:20.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:20.230+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:20.230+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:20.230+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:20.230+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:21.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:20.801+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:21.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:20.801+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:21.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:20.801+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:21.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:20.801+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:21.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:21.510+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:21.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:21.510+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:21.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:21.510+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:21.275+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:21.275+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:21.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:21.275+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:21.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:21.275+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:22.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:21.915+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3427 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:21.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:21.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:22.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:21.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:22.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:21.785+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:22.272+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:22.272+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:22.272+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:22.272+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:22.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:22.521+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:22.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:22.521+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:22.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:22.521+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:23.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:22.792+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:23.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:22.792+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:23.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:22.792+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:23.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:22.792+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:23.478+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:23.478+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:23.478+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:23.288+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:23.288+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:23.288+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:23.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:23.288+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:23.793+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:23.793+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:23.793+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:23.793+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:24.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:24.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:24.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:24.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:24.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:24.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:24.292+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:24.292+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:24.292+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:24.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:24.292+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:24.744+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:24.744+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:24.744+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:24.744+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:25.332+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:25.332+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:25.332+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:25.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:25.332+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:25.499+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:25.499+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:25.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:25.499+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:25.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:25.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:25.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:25.785+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:26.290+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:26.290+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:26.290+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:26.291+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:26.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:26.523+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:26.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:26.523+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:26.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:26.523+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:27.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:26.917+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3483 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:26.764+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:26.764+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:26.764+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:26.764+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:27.491 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:27.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:27.491 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:27.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:27.492 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:27.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:27.492 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:27.333+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:27.776 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:27.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:27.776 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:27.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:27.776 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:27.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:28.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:27.788+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:28.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:27.788+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:28.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:27.788+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:28.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:27.788+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:28.354+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:28.354+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:28.354+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:28.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:28.355+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:28.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:28.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:28.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:28.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:28.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:28.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:29.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:28.823+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:29.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:28.823+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:29.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:28.823+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:29.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:28.824+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:29.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:29.370+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:29.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:29.370+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:29.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:29.370+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:29.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:29.370+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:29.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:29.529+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:29.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:29.529+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:29.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:29.529+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:29.833+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:29.833+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:29.833+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:29.833+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:30.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:30.380+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:30.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:30.380+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:30.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:30.380+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:30.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:30.380+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:30.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:30.566+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:30.566+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:30.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:30.566+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:30.833+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:30.833+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:30.833+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:30.833+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:31.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:31.394+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:31.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:31.394+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:31.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:31.394+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:31.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:31.394+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:31.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:31.578+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:31.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:31.578+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:31.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:31.578+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:32.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:31.918+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3541 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:31.800+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:31.800+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:32.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:31.800+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:32.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:31.800+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:32.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:32.400+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:32.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:32.400+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:32.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:32.400+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:32.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:32.400+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:32.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:32.597+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:32.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:32.597+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:32.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:32.597+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:33.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:32.820+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:33.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:32.820+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:33.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:32.820+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:33.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:32.820+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:33.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:33.414+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:33.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:33.414+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:33.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:33.414+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:33.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:33.414+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:33.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:33.644+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:33.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:33.644+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:33.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:33.644+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:34.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:33.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:34.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:33.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:34.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:33.859+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:34.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:33.859+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:34.382+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:34.382+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:34.382+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:34.383+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:34.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:34.655+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:34.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:34.655+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:34.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:34.655+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:34.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:34.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:34.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:34.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:34.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:34.879+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:34.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:34.879+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:35.656 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:35.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:35.656 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:35.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:35.656 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:35.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:35.656 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:35.367+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:35.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:35.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:35.891+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:35.891+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:35.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:35.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:35.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:35.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:35.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:35.668+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:36.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:36.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:36.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:36.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:36.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:36.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:36.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:36.333+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:36.907 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:36.655+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:36.907 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:36.655+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:36.907 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:36.655+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:36.907 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:36.862+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:36.907 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:36.862+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:36.907 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:36.862+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:36.908 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:36.862+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:37.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:36.919+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3596 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:37.589 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:37.286+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:37.589 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:37.286+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:37.589 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:37.286+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:37.589 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:37.286+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:37.883 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:37.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:37.883 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:37.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:37.883 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:37.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:38.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:37.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:38.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:37.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:38.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:37.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:38.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:37.895+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:38.578 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:38.324+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:38.578 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:38.324+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:38.578 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:38.324+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:38.578 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:38.325+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:38.869 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:38.590+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:38.869 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:38.590+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:38.869 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:38.590+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:38.881+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:38.881+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:38.881+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:38.881+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:39.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:39.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:39.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:39.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:39.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:39.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:39.603 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:39.338+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:39.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:39.615+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:39.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:39.615+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:39.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:39.615+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:39.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:39.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:39.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:39.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:39.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:39.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:39.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:39.835+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:40.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:40.363+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:40.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:40.363+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:40.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:40.363+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:40.610 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:40.363+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:40.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:40.623+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:40.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:40.623+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:40.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:40.623+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:40.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:40.791+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:40.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:40.791+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:40.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:40.791+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:40.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:40.791+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:41.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:41.385+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:41.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:41.385+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:41.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:41.385+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:41.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:41.385+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:41.958 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:41.920+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3653 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:41.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:41.671+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:41.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:41.671+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:41.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:41.671+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:41.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:41.823+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:41.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:41.823+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:41.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:41.823+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:41.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:41.823+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:42.637 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:42.370+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:42.638 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:42.370+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:42.638 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:42.370+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:42.638 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:42.370+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:42.700+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:42.700+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:42.700+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:42.837+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:42.837+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:42.837+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:42.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:42.837+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:43.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:43.378+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:43.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:43.378+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:43.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:43.378+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:43.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:43.378+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:43.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:43.698+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:43.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:43.698+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:43.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:43.698+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:43.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:43.873+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:43.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:43.873+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:43.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:43.873+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:43.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:43.873+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:44.678 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:44.401+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:44.678 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:44.401+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:44.678 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:44.401+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:44.678 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:44.401+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:44.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:44.690+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:44.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:44.690+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:44.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:44.690+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:44.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:44.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:44.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:44.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:44.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:44.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:44.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:44.835+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:45.700 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:45.410+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:45.700 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:45.410+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:45.700 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:45.410+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:45.700 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:45.410+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:45.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:45.712+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:45.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:45.712+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:45.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:45.712+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:45.814+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:45.814+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:45.814+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:45.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:45.814+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:46.436+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:46.436+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:46.436+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:46.436+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:47.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:46.921+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3710 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:46.721+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:46.721+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:47.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:46.721+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:46.858+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:46.858+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:46.858+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:46.858+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:47.686 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:47.387+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:47.687 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:47.387+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:47.687 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:47.387+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:47.687 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:47.387+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:47.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:47.748+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:47.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:47.748+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:47.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:47.748+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:47.884+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:47.884+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:47.884+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:47.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:47.884+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:48.702 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:48.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:48.702 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:48.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:48.703 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:48.390+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:48.703 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:48.390+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:48.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:48.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:48.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:48.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:48.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:48.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:48.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:48.886+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:48.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:48.886+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:48.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:48.886+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:48.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:48.886+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:49.679 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:49.428+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:49.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:49.428+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:49.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:49.428+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:49.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:49.428+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:49.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:49.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:49.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:49.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:49.895+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:49.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:49.895+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:49.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:49.692+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:49.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:49.692+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:49.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:49.692+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:50.694 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:50.437+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:50.694 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:50.437+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:50.694 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:50.437+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:50.694 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:50.437+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:50.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:50.864+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:50.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:50.864+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:50.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:50.864+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:50.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:50.864+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:50.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:50.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:50.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:50.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:50.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:50.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:51.442+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:51.442+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:51.442+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:51.443+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:52.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:51.921+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3769 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:52.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:51.720+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:52.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:51.720+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:52.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:51.720+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:52.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:51.825+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:52.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:51.825+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:52.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:51.825+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:52.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:51.825+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:52.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:52.396+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:52.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:52.396+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:52.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:52.396+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:52.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:52.396+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:53.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:52.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:53.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:52.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:53.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:52.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:53.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:52.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:53.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:52.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:53.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:52.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:53.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:52.790+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:53.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:53.415+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:53.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:53.415+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:53.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:53.415+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:53.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:53.415+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:54.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:53.784+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:54.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:53.784+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:54.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:53.784+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:53.799+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:53.799+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:53.799+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:54.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:53.799+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:54.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:54.433+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:54.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:54.433+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:54.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:54.433+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:54.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:54.433+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:55.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:54.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:55.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:54.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:55.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:54.792+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:55.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:54.749+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:55.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:54.749+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:55.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:54.749+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:55.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:54.749+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:55.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:55.408+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:55.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:55.408+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:55.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:55.408+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:55.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:55.408+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:56.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:55.753+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:56.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:55.753+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:56.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:55.753+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:56.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:55.791+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:56.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:55.791+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:56.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:55.791+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:56.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:55.791+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:56.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:56.458+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:56.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:56.458+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:56.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:56.458+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:56.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:56.458+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:57.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:08:56.921+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3825 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:08:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:56.749+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:56.749+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:56.749+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:56.830+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:56.830+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:56.830+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:57.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:56.830+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:57.769 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:57.484+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:57.769 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:57.484+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:57.769 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:57.484+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:57.770 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:57.484+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:58.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:57.780+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:58.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:57.780+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:58.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:57.780+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:58.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:57.843+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:58.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:57.843+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:58.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:57.843+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:58.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:57.843+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:58.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:58.511+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:58.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:58.511+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:58.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:58.511+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:58.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:58.511+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:58.814 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:58.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:58.814 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:58.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:58.814 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:58.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:08:59.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:58.824+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:59.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:58.824+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:59.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:58.824+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:08:59.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:58.824+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:08:59.758 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:59.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:59.758 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:59.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:59.758 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:59.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:08:59.759 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:08:59.472+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:00.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:59.770+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:00.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:59.770+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:00.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:08:59.770+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:00.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:59.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:59.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:59.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:00.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:08:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:08:59.782+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:00.464+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:00.464+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:00.464+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:00.464+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:01.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:00.815+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:01.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:00.815+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:01.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:00.815+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:00.803+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:00.803+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:00.803+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:01.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:00.803+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:01.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:01.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:01.449+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:01.449+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:02.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:01.922+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3884 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:02.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:01.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:02.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:01.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:02.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:01.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:01.821+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:01.821+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:01.821+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:02.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:01.821+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:02.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:02.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:02.409+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:02.409+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:03.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:02.826+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:03.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:02.826+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:03.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:02.826+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:03.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:02.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:02.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:02.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:03.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:02.782+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:03.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:03.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:03.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:03.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:03.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:03.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:03.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:03.406+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:04.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:03.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:04.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:03.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:04.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:03.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:03.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:03.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:03.785+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:04.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:03.785+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:04.362+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:04.362+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:04.362+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:04.362+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:05.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:04.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:05.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:04.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:05.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:04.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:05.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:04.778+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:04.778+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:04.778+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:05.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:04.778+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:05.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:05.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:05.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:05.338+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:06.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:05.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:06.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:05.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:06.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:05.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:06.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:05.824+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:05.824+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:05.824+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:06.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:05.824+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:06.340+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:06.340+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:06.340+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:06.340+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:07.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:06.923+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3941 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:07.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:06.923+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:07.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:06.923+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:07.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:06.923+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:06.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:06.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:07.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:06.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:07.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:06.834+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:07.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:07.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:07.367+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:07.367+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:08.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:07.882+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:08.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:07.882+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:08.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:07.882+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:07.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:07.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:07.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:08.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:07.834+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:08.416+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:08.416+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:08.416+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:08.416+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:09.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:08.864+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:09.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:08.864+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:09.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:08.864+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:09.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:08.808+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:09.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:08.808+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:09.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:08.808+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:09.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:08.808+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:09.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:09.432+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:09.432+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:09.432+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:09.432+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:10.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:09.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:10.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:09.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:10.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:09.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:10.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:09.765+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:09.765+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:09.765+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:10.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:09.765+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:10.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:10.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:10.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:10.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:10.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:10.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:10.741 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:10.472+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:11.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:10.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:11.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:10.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:11.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:10.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:11.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:10.753+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:11.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:10.753+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:11.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:10.753+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:11.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:10.753+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:11.902 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:11.709+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:11.902 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:11.709+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:11.903 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:11.709+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:11.903 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:11.709+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:11.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:11.514+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:11.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:11.514+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:11.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:11.514+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:11.904 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:11.514+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:12.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:11.924+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 3999 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:12.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:11.914+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:12.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:11.914+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:12.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:11.914+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:12.932 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:12.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:12.932 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:12.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:12.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:12.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:12.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:12.549+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:12.933 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:12.738+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:12.933 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:12.738+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:12.933 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:12.738+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:12.934 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:12.738+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:13.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:12.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:13.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:12.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:13.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:12.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:13.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:13.712+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:13.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:13.712+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:13.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:13.712+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:13.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:13.712+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:13.554+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:13.554+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:13.554+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:13.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:13.554+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:14.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:13.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:14.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:13.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:14.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:13.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:14.946 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:14.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:14.946 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:14.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:14.946 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:14.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:14.947 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:14.546+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:14.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:14.737+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:14.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:14.737+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:14.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:14.737+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:14.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:14.737+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:15.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:14.958+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:15.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:14.958+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:15.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:14.958+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:15.908 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:15.541+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:15.909 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:15.541+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:15.909 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:15.541+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:15.909 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:15.541+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:15.909 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:15.706+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:15.910 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:15.706+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:15.910 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:15.706+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:15.910 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:15.706+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:16.184 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:15.920+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:16.184 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:15.920+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:16.184 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:15.920+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:16.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:16.721+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:16.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:16.721+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:16.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:16.721+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:16.915 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:16.721+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:16.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:16.516+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:16.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:16.516+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:16.916 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:16.516+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:16.916 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:16.516+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:17.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:16.926+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4054 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:17.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:16.945+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:17.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:16.945+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:17.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:16.945+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:17.914 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:17.521+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:17.914 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:17.521+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:17.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:17.521+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:17.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:17.521+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:17.915 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:17.740+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:17.915 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:17.740+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:17.915 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:17.740+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:17.915 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:17.741+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:18.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:17.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:18.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:17.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:18.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:17.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:18.888 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:18.695+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:18.695+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:18.695+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:18.696+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:18.515+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:18.515+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:18.515+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:18.889 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:18.515+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:19.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:18.901+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:19.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:18.901+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:19.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:18.901+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:19.679 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:19.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:19.679 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:19.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:19.679 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:19.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:19.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:19.467+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:19.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:19.867+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:19.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:19.867+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:19.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:19.867+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:19.691+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:19.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:19.691+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:19.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:19.691+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:19.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:19.692+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:20.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:20.652+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:20.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:20.652+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:20.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:20.652+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:20.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:20.652+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:20.424+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:20.424+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:20.424+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:20.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:20.425+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:21.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:20.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:21.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:20.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:21.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:20.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:21.674+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:21.674+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:21.674+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:21.675+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:21.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:21.455+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:21.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:21.455+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:21.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:21.455+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:21.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:21.455+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:22.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:21.927+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4112 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:22.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:21.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:22.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:21.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:22.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:21.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:22.683 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:22.481+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:22.683 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:22.481+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:22.683 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:22.481+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:22.683 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:22.481+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:22.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:22.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:22.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:22.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:22.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:22.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:22.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:22.695+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:22.695+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:22.695+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:22.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:22.695+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:23.463+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:23.463+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:23.463+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:23.463+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:24.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:23.776+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:24.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:23.776+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:24.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:23.776+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:24.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:23.730+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:24.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:23.730+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:24.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:23.730+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:24.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:23.730+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:24.743 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:24.500+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:24.743 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:24.500+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:24.743 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:24.500+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:24.743 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:24.500+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:25.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:24.796+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:25.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:24.796+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:25.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:24.796+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:24.755+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:24.755+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:24.755+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:25.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:24.755+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:25.781 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:25.508+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:25.781 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:25.508+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:25.781 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:25.508+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:25.782 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:25.508+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:25.782 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:25.739+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:25.782 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:25.739+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:25.782 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:25.739+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:25.782 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:25.739+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:26.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:25.794+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:25.794+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:25.794+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:26.699 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:26.460+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:26.699 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:26.460+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:26.699 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:26.460+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:26.700 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:26.460+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:26.958 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:26.928+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4167 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:26.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:26.827+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:26.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:26.827+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:26.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:26.827+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:26.712+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:26.712+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:26.712+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:26.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:26.712+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:27.817 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:27.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:27.817 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:27.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:27.817 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:27.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:27.817 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:27.671+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:27.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:27.472+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:27.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:27.473+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:27.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:27.473+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:27.818 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:27.473+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:28.080 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:27.830+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:28.080 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:27.830+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:28.080 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:27.830+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:28.859 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:28.475+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:28.859 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:28.476+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:28.859 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:28.476+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:28.860 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:28.476+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:28.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:28.667+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:28.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:28.667+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:28.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:28.667+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:28.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:28.667+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:29.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:28.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:28.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:28.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:29.859 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:29.685+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:29.685+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:29.685+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:29.685+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:29.480+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:29.480+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:29.480+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:29.860 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:29.480+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:30.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:29.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:30.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:29.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:30.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:29.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:30.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:30.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:30.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:30.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:30.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:30.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:30.661+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:30.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:30.452+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:30.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:30.452+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:30.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:30.452+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:30.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:30.452+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:30.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:30.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:30.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:31.669 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:31.414+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:31.669 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:31.414+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:31.669 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:31.414+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:31.669 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:31.414+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:31.958 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:31.928+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4226 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:31.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:31.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:31.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:31.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:31.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:31.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:31.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:31.682+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:31.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:31.682+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:31.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:31.682+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:31.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:31.682+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:32.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:32.372+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:32.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:32.372+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:32.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:32.372+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:32.660 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:32.372+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:32.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:32.918+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:32.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:32.918+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:32.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:32.918+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:32.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:32.672+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:32.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:32.672+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:32.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:32.672+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:32.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:32.672+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:33.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:33.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:33.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:33.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:33.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:33.333+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:33.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:33.333+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:33.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:33.874+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:33.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:33.874+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:33.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:33.874+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:33.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:33.673+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:33.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:33.673+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:33.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:33.673+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:33.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:33.674+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:34.642 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:34.351+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:34.351+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:34.351+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:34.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:34.351+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:34.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:34.886+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:34.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:34.886+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:34.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:34.886+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:34.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:34.655+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:34.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:34.655+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:34.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:34.655+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:34.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:34.655+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:35.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:35.374+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:35.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:35.374+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:35.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:35.374+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:35.685 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:35.374+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:35.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:35.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:35.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:35.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:35.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:35.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:35.698+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:35.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:35.698+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:35.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:35.698+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:35.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:35.698+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:36.371+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:36.371+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:36.371+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:36.371+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:37.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:36.929+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4283 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:36.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:36.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:36.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:36.741+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:36.741+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:36.741+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:36.741+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:37.358+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:37.358+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:37.358+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:37.358+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:38.178 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:37.756+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:38.178 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:37.756+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:38.178 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:37.756+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:38.178 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:37.756+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:38.179 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:37.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:38.179 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:37.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:38.179 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:37.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:38.309+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:38.309+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:38.309+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:38.309+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:38.943+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:39.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:38.943+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:39.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:38.943+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:38.770+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:38.770+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:39.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:38.770+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:39.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:38.770+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:39.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:39.327+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:39.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:39.327+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:39.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:39.327+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:39.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:39.327+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:40.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:39.922+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:40.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:39.922+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:40.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:39.922+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:39.752+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:39.752+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:39.752+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:39.752+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:40.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:40.339+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:40.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:40.339+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:40.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:40.339+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:40.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:40.339+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:41.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:40.915+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:41.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:40.915+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:41.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:40.915+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:41.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:40.794+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:41.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:40.794+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:41.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:40.794+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:41.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:40.794+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:41.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:41.316+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:41.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:41.316+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:41.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:41.316+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:41.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:41.316+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:42.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:41.930+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4342 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:42.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:41.924+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:42.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:41.924+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:42.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:41.924+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:41.784+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:41.784+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:41.784+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:41.784+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:42.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:42.293+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:42.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:42.293+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:42.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:42.293+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:42.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:42.293+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:43.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:42.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:43.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:42.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:43.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:42.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:43.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:42.764+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:43.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:42.764+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:43.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:42.764+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:43.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:42.764+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:43.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:43.278+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:43.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:43.278+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:43.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:43.278+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:43.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:43.278+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:44.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:43.966+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:44.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:43.966+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:44.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:43.966+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:44.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:43.808+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:43.808+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:43.808+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:44.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:43.808+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:44.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:44.265+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:44.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:44.265+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:44.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:44.265+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:44.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:44.265+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:45.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:44.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:45.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:44.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:45.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:44.979+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:45.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:44.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:45.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:44.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:45.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:44.834+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:45.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:44.834+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:45.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:45.311+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:45.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:45.311+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:45.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:45.311+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:45.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:45.311+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:46.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:46.000+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:46.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:46.000+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:46.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:46.000+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:45.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:45.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:45.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:46.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:45.813+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:46.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:46.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:46.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:46.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:46.291+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:47.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:46.930+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4400 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:47.006+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:47.006+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:47.006+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:46.797+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:46.797+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:46.797+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:47.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:46.797+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:47.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:47.335+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:47.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:47.335+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:47.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:47.335+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:47.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:47.335+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:48.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:47.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:48.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:47.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:48.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:47.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:48.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:47.846+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:48.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:48.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:48.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:48.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:48.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:48.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:48.321+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:48.321+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:48.321+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:48.321+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:49.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:49.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:49.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:49.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:49.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:49.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:49.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:48.860+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:49.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:48.860+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:49.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:48.860+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:49.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:48.860+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:49.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:49.287+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:49.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:49.287+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:49.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:49.287+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:49.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:49.287+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:50.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:50.015+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:50.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:50.015+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:50.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:50.015+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:50.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:49.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:50.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:49.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:50.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:49.857+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:50.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:49.857+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:50.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:50.320+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:50.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:50.320+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:50.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:50.320+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:50.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:50.320+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:51.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:50.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:51.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:50.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:51.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:50.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:51.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:50.827+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:51.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:50.827+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:51.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:50.827+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:51.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:50.827+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:51.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:19.518847+0000 front 2024-04-15T22:59:19.509603+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:51.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:19.510900+0000 front 2024-04-15T22:59:19.509938+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:51.344+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:19.503406+0000 front 2024-04-15T22:59:19.504880+0000 (oldest deadline 2024-04-15T22:59:44.202961+0000) 2024-04-15T23:09:51.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:51.345+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:52.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:51.931+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4459 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:52.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:52.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:52.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:52.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:52.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:52.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:51.862+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:51.862+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:51.862+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:51.862+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:52.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:52.348+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:53.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:53.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:53.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:52.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:52.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:52.863+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:53.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:52.863+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:53.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:53.394+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:54.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:54.032+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:54.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:54.032+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:54.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:54.032+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:54.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:53.866+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:54.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:53.866+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:54.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:53.866+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:54.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:53.866+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:54.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:54.369+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:55.062 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:54.820+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:55.062 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:54.820+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:55.062 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:54.820+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:55.062 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:54.820+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:55.340 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:55.074+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:55.340 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:55.074+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:55.340 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:55.074+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:55.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:55.352+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:56.108 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:55.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:56.108 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:55.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:56.108 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:55.846+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:56.108 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:55.846+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:56.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:56.318+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:56.120+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:56.120+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:56.120+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:57.150 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:09:56.932+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4516 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:09:57.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:56.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:57.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:56.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:57.150 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:56.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:57.151 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:56.805+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:57.350+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:57.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:57.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:57.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:58.177 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:57.795+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:58.177 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:57.795+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:58.177 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:57.795+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:58.177 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:57.795+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:58.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:58.395+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:58.189+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:58.189+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:58.189+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:59.168 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:58.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:59.169 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:58.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:59.169 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:58.835+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:09:59.169 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:58.835+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:09:59.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:59.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:59.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:09:59.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:09:59.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:09:59.358+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:00.139 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:59.819+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:00.139 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:59.819+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:00.139 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:59.819+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:00.139 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:09:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:09:59.819+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:00.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:00.366+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:00.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:22.785555+0000 front 2024-04-15T22:59:22.785631+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:10:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:00.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:22.785714+0000 front 2024-04-15T22:59:22.785609+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:10:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:00.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:22.785659+0000 front 2024-04-15T22:59:22.785642+0000 (oldest deadline 2024-04-15T22:59:46.885613+0000) 2024-04-15T23:10:01.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:00.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:00.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:00.813+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:00.813+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:01.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:01.416+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:02.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:01.933+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4575 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:01.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:01.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:01.804+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:01.805+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:02.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:02.393+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:03.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:02.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:03.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:02.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:03.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:02.782+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:03.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:02.782+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:03.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:03.347+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:04.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:03.794+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:03.794+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:03.794+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:04.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:03.794+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:04.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:04.303+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:05.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:04.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 since back 2024-04-15T22:59:21.006261+0000 front 2024-04-15T22:59:21.006275+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:05.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:04.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 since back 2024-04-15T22:59:21.006288+0000 front 2024-04-15T22:59:21.006299+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:05.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:04.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 since back 2024-04-15T22:59:21.006336+0000 front 2024-04-15T22:59:21.006327+0000 (oldest deadline 2024-04-15T22:59:45.106169+0000) 2024-04-15T23:10:05.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:04.790+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:05.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:05.332+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:06.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:05.768+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:06.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:06.316+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:07.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:06.935+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4632 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:07.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:06.782+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:07.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:07.361+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:08.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:07.776+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:08.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:08.381+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:09.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:08.738+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:09.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:09.377+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:10.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:09.773+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:10.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:10.387+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:11.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:10.743+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:11.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:11.345+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:12.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:11.936+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4690 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:11.746+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:12.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:12.312+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:13.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:12.793+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:13.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:13.356+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:13.356+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:13.356+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:13.356+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:14.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:13.837+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:14.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:14.392+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:14.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:14.392+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:14.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:14.392+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:14.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:14.392+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:15.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:14.862+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:15.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:15.372+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:15.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:15.372+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:15.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:15.372+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:15.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:15.372+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:16.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:15.888+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:16.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:16.342+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:16.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:16.342+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:16.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:16.342+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:16.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:16.342+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:17.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:16.937+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4746 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:17.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:16.902+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:17.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:17.380+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:17.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:17.380+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:17.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:17.380+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:17.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:17.380+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:18.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:17.950+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:18.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:18.384+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:18.384+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:18.384+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:18.384+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:19.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:18.972+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:19.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:19.413+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:19.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:19.413+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:19.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:19.413+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:19.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:19.414+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:20.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:19.969+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:20.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:20.456+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:20.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:20.456+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:20.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:20.456+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:20.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:20.456+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:21.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:21.017+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:21.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:21.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:21.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:21.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:21.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:21.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:21.925 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:21.477+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:21.926 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:21.477+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:21.926 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:21.477+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:21.926 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:21.477+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:22.208 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:21.938+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4804 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:22.033+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:22.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:22.141+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:22.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:22.141+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:22.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:22.141+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:22.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:22.439+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:22.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:22.439+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:22.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:22.439+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:22.680 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:22.439+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:23.438 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:22.998+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:23.442 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:23.156+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:23.442 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:23.156+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:23.442 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:23.156+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:23.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:23.451+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:23.451+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:23.451+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:23.451+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:24.007+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:24.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:24.147+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:24.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:24.147+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:24.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:24.147+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:24.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:24.494+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:24.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:24.494+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:24.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:24.494+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:24.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:24.494+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:25.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:24.975+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:25.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:25.196+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:25.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:25.196+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:25.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:25.196+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:25.752 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:25.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:25.752 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:25.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:25.752 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:25.485+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:25.752 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:25.485+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:26.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:25.965+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:26.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:26.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:26.165+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:26.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:26.497+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:26.916 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:26.497+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:26.916 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:26.497+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:26.916 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:26.497+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:27.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:26.939+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4860 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:26.928+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:27.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:27.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:27.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:27.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:27.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:27.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:27.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:27.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:27.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:27.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:27.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:27.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:27.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:27.503+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:28.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:27.951+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:28.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:28.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:28.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:28.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:28.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:28.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:28.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:28.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:28.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:28.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:28.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:28.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:28.934 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:28.544+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:29.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:28.946+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:29.143+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:29.143+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:29.143+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:29.901 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:29.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:29.901 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:29.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:29.901 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:29.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:29.901 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:29.588+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:30.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:30.102+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:30.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:30.102+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:30.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:30.102+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:29.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:29.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:29.914+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:30.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:29.914+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:30.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:30.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:30.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:30.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:30.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:30.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:30.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:30.633+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:31.075+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:31.075+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:31.075+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:30.958+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:30.958+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:30.958+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:30.958+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:31.927 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:31.669+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:31.928 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:31.669+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:31.928 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:31.669+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:31.928 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:31.669+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:32.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:31.940+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4918 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:32.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:32.099+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:32.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:32.099+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:32.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:32.099+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:32.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:32.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:32.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:32.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:32.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:32.000+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:32.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:32.662+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:32.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:32.662+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:32.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:32.662+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:32.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:32.662+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:33.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:33.136+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:33.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:33.136+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:33.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:33.136+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:33.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:33.020+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:33.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:33.020+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:33.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:33.020+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:33.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:33.021+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:33.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:33.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:33.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:33.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:33.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:33.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:33.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:33.670+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:34.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:34.104+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:34.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:34.104+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:34.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:34.104+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:34.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:34.055+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:34.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:34.055+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:34.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:34.055+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:34.462 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:34.055+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:34.641+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:34.641+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:34.641+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:34.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:34.641+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:35.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:35.067+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:35.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:35.067+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:35.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:35.067+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:35.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.035+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:35.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.035+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:35.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.035+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:35.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.035+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:35.630+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:35.630+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:35.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:35.630+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:35.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:35.630+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:36.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:36.085+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:36.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:36.085+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:36.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:36.085+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:36.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:36.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:36.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.989+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:36.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:35.989+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:36.928 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:36.660+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:36.929 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:36.660+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:36.929 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:36.660+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:36.929 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:36.660+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:37.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:36.940+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 4974 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:37.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:37.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:37.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:37.004+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:37.004+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:37.004+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:37.004+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:37.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:37.615+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:37.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:37.615+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:37.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:37.615+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:37.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:37.615+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:38.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:38.003+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:38.003+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:38.003+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:38.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:38.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:38.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:38.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:38.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:38.041+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:38.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:38.041+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:38.943 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:38.631+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:38.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:38.631+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:38.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:38.631+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:38.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:38.631+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:38.956+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:38.956+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:39.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:38.956+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:39.055+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:39.055+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:39.055+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:39.055+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:39.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:39.647+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:39.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:39.647+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:39.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:39.647+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:39.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:39.647+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:40.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:39.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:40.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:39.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:40.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:39.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:40.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:40.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:40.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:40.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:40.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:40.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:40.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:40.058+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:40.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:40.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:40.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:40.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:40.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:40.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:40.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:40.672+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:41.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:40.996+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:41.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:40.996+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:41.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:40.996+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:41.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:41.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:41.103+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:41.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:41.103+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:41.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:41.941+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5032 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:41.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:41.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:41.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:41.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:41.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:41.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:41.963 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:41.721+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:42.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:41.975+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:42.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:41.975+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:42.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:41.975+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:42.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:42.117+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:42.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:42.117+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:42.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:42.117+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:42.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:42.117+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:43.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:42.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:43.154 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:42.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:43.154 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:42.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:43.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:42.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:43.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:42.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:43.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:42.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:43.155 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:42.747+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:43.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:43.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:43.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:43.166+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:44.001 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:43.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:44.001 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:43.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:44.002 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:43.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:44.002 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:43.724+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:44.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:44.013+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:44.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:44.013+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:44.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:44.013+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:44.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:44.203+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:44.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:44.203+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:44.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:44.203+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:44.203+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:44.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:44.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:44.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:44.689+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:45.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:45.037+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:45.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:45.037+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:45.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:45.037+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:45.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:45.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:45.211+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:45.211+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:45.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:45.698+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:45.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:45.698+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:45.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:45.698+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:45.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:45.698+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:46.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:46.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:46.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:46.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:46.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:46.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:46.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:46.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:46.256+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:46.256+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:46.931 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:46.658+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:46.931 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:46.658+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:46.931 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:46.658+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:46.931 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:46.658+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:47.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:46.942+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5088 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:47.095+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:47.095+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:47.095+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:47.668 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:47.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:47.669 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:47.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:47.669 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:47.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:47.669 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:47.280+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:47.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:47.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:47.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:47.680+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:48.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:48.122+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:48.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:48.122+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:48.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:48.122+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:48.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:48.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:48.282+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:48.282+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:49.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:48.722+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:49.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:48.722+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:49.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:48.722+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:49.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:48.722+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:49.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:49.091+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:49.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:49.091+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:49.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:49.091+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:49.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:49.244+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:49.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:49.244+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:49.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:49.244+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:49.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:49.244+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:50.040 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:49.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:50.041 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:49.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:50.041 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:49.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:50.041 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:49.748+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:50.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:50.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:50.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:50.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:50.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:50.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:50.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:50.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:50.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:50.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:50.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:50.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:50.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:50.230+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:51.176 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:51.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:51.176 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:51.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:51.176 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:51.025+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:51.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:50.791+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:51.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:50.791+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:51.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:50.791+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:51.177 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:50.791+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:51.188+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:51.188+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:51.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:51.188+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:51.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:51.188+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:52.030 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:51.943+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5146 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:52.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:51.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:52.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:51.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:52.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:51.790+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:52.030 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:51.790+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:52.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:52.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:52.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:52.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:52.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:52.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:52.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:52.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:52.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:52.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:52.239+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:52.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:52.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:52.773+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:52.773+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:53.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:53.062+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:53.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:53.062+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:53.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:53.062+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:53.198+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:53.198+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:53.198+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:53.198+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:54.047 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:53.764+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:54.048 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:53.764+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:54.048 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:53.764+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:54.048 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:53.764+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:54.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:54.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:54.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:54.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:54.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:54.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:54.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:54.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:54.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:54.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:54.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:54.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:54.230+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:55.037 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:54.751+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:55.037 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:54.751+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:55.037 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:54.751+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:55.037 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:54.751+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:55.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:55.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:55.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:55.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:55.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:55.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:55.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:55.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:55.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:55.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:55.195+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:55.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:55.195+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:56.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:55.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:56.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:55.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:56.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:55.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:56.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:55.748+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:56.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:56.092+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:56.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:56.092+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:56.092+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:56.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:56.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:56.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:56.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:56.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:56.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:56.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:56.193+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:57.100 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:10:56.945+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5202 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:10:57.100 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:56.770+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:57.100 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:56.770+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:57.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:56.770+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:57.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:56.770+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:57.113+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:57.113+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:57.113+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:57.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:57.161+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:57.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:57.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:57.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:57.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:57.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:57.162+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:58.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:57.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:58.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:57.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:58.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:57.796+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:58.101 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:57.797+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:58.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:58.113+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:58.113+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:58.113+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:58.194+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:58.194+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:58.194+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:58.194+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:59.096 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:58.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:59.097 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:58.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:59.097 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:58.814+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:10:59.097 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:58.814+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:10:59.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:59.109+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:59.109+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:10:59.109+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:10:59.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:59.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:59.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:59.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:59.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:59.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:10:59.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:10:59.171+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:00.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:59.775+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:00.051 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:59.775+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:00.051 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:59.775+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:00.051 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:10:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:10:59.775+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:00.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:00.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:00.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:00.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:00.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:00.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:00.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:00.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:00.200+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:00.200+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:01.068 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:00.793+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:01.069 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:00.793+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:01.069 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:00.793+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:01.069 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:00.793+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:01.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:01.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:01.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:01.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:01.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:01.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:01.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:01.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:01.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:01.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:01.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:01.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:01.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:01.186+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:02.173 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:01.946+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5260 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:02.173 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:02.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:02.173 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:02.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:02.173 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:02.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:02.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:01.842+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:02.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:01.842+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:02.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:01.842+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:02.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:01.842+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:02.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:02.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:02.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:02.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:02.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:02.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:02.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:02.186+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:03.167 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:03.044+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:03.167 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:03.044+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:03.167 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:03.044+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:03.167 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:02.830+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:03.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:02.830+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:03.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:02.830+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:03.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:02.830+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:03.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:03.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:03.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:03.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:03.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:03.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:03.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:03.180+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:04.196 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:04.028+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:04.197 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:04.028+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:04.197 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:04.028+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:04.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:03.839+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:04.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:03.839+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:04.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:03.839+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:04.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:03.840+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:04.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:04.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:04.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:04.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:04.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:04.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:04.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:04.209+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:05.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:05.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:05.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:05.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:05.187 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:05.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:05.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:04.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:05.187 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:04.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:05.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:04.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:05.188 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:04.855+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:05.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:05.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:05.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:05.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:05.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:05.199+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:05.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:05.200+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:06.163 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:06.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:06.163 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:06.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:06.163 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:06.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:06.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:05.893+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:06.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:05.893+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:06.164 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:05.893+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:06.164 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:05.893+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:06.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:06.176+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:06.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:06.176+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:06.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:06.176+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:06.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:06.176+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:07.193 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:06.947+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5316 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:07.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:07.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:07.066+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:06.863+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:06.863+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:06.863+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:07.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:06.863+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:07.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:07.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:07.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:07.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:07.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:07.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:07.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:07.205+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:08.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:07.834+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:08.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:07.834+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:08.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:07.834+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:08.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:07.834+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:08.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:08.084+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:08.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:08.084+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:08.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:08.084+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:08.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:08.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:08.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:08.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:08.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:08.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:08.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:08.230+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:09.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:09.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:09.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:09.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:09.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:09.050+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:09.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:08.843+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:09.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:08.843+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:09.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:08.843+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:09.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:08.843+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:09.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:09.202+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:09.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:09.202+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:09.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:09.202+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:09.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:09.202+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:10.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:10.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:10.154 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:10.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:10.154 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:10.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:10.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:09.860+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:10.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:09.860+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:10.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:09.860+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:10.154 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:09.860+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:10.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:10.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:10.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:10.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:10.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:10.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:10.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:10.166+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:11.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:11.119+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:11.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:11.119+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:11.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:11.119+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:11.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:11.119+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:11.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:11.056+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:11.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:11.056+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:11.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:11.056+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:11.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:10.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:11.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:10.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:11.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:10.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:11.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:10.869+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:12.138 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:11.948+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5374 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:12.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:11.844+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:12.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:11.844+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:12.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:11.844+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:12.139 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:11.844+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:12.139 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:12.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:12.140 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:12.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:12.140 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:12.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:12.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:12.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:12.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:12.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:12.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:12.151+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:12.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:12.151+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:13.151 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:13.107+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:13.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:13.107+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:13.152 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:13.107+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:13.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:12.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:13.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:12.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:13.152 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:12.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:13.153 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:12.874+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:13.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:13.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:13.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:13.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:13.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:13.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:13.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:13.165+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:14.108 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:14.067+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:14.108 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:14.067+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:14.108 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:14.067+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:14.108 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:13.831+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:14.109 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:13.831+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:14.109 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:13.831+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:14.109 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:13.831+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:14.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:14.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:14.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:14.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:14.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:14.160+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:14.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:14.161+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:15.143 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:15.064+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:15.143 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:15.064+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:15.143 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:15.064+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:15.144 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:14.854+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:15.144 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:14.854+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:15.144 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:14.854+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:15.144 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:14.855+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:15.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:15.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:15.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:15.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:15.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:15.156+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:15.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:15.156+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:16.172 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:16.027+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:16.172 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:16.027+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:16.173 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:16.027+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:16.173 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:15.843+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:16.173 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:15.843+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:16.173 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:15.843+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:16.173 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:15.843+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:16.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:16.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:16.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:16.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:16.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:16.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:16.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:16.185+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:17.179 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:16.949+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5430 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:17.180 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:17.011+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:17.180 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:17.011+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:17.180 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:17.011+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:17.180 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:16.880+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:17.181 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:16.880+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:17.181 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:16.880+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:17.181 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:16.880+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:17.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:17.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:17.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:17.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:17.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:17.191+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:17.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:17.191+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:17.911+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:17.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:17.911+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:17.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:17.911+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:17.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:17.911+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:18.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:18.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:18.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:18.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:18.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:18.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:18.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:18.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:18.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:18.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:18.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:18.162+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:18.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:18.162+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:19.131 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:19.009+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:19.141 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:19.009+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:19.141 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:19.009+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:19.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:18.870+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:19.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:18.870+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:19.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:18.870+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:19.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:18.870+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:19.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:19.145+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:19.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:19.145+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:19.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:19.145+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:19.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:19.145+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:20.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:20.125+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:20.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:20.125+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:20.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:20.125+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:20.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:20.125+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:20.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:20.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:20.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:20.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:20.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:20.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:20.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:19.907+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:20.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:19.907+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:20.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:19.907+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:20.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:19.907+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:21.016 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:20.880+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:21.016 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:20.880+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:21.016 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:20.880+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:21.016 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:20.880+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:21.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:21.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:21.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:21.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:21.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:21.029+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:21.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:21.076+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:21.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:21.076+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:21.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:21.076+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:21.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:21.077+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:22.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:21.950+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5488 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:22.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:22.061+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:22.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:22.061+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:22.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:22.061+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:22.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:21.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:22.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:21.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:22.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:21.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:22.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:21.894+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:22.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:22.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:22.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:22.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:22.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:22.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:22.210 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:22.044+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:23.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:23.082+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:23.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:23.082+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:23.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:23.082+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:23.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:23.082+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:23.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:23.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:23.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:23.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:23.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:23.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:22.908+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:22.908+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:22.908+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:22.908+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:24.074 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:23.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:24.074 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:23.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:24.075 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:23.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:24.075 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:23.912+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:24.075 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:23.912+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:24.075 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:23.912+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:24.075 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:23.912+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:24.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:24.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:24.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:24.086+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:24.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:24.086+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:25.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:25.045+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:25.045+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:25.045+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:25.045+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:25.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:24.972+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:25.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:24.972+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:25.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:24.972+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:25.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:24.862+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:25.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:24.862+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:25.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:24.862+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:25.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:24.862+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:26.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:26.027+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:26.027+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:26.027+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:26.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:26.027+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:26.007+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:26.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:26.007+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:26.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:26.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:26.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:25.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:26.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:25.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:26.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:25.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:26.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:25.867+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:27.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:26.950+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5544 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:27.011+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:27.011+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:27.011+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:27.011+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:27.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:27.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:27.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:27.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:27.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:27.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:27.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:26.910+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:27.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:26.910+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:27.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:26.910+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:27.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:26.910+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:28.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:28.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:28.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:28.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.044+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:28.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:27.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:28.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:27.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:28.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:27.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:27.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:27.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:27.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:28.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:27.886+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:29.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:29.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:29.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:29.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:28.997+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:29.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:29.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:29.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:29.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:29.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:28.890+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:29.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:28.890+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:29.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:28.890+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:29.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:28.890+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:30.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:30.043+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:30.043+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:30.043+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:30.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:30.043+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:30.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:30.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:30.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:30.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:30.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:30.046+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:29.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:29.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:29.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:30.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:29.900+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:31.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:31.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:31.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:31.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:31.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:31.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:31.064+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:31.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:31.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:31.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:31.081+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:31.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:30.871+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:31.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:30.871+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:31.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:30.871+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:31.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:30.871+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:32.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:31.951+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5602 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:32.074+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:32.074+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:32.074+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:32.074+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:32.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:32.110+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:32.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:32.110+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:32.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:32.110+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:32.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:31.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:32.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:31.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:32.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:31.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:32.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:31.866+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:33.132 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:32.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:33.132 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:32.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:33.132 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:32.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:33.132 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:32.866+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:33.132 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:33.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:33.133 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:33.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:33.133 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:33.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:33.133 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:33.097+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:33.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:33.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:33.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:33.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:33.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:33.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:34.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:33.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:34.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:33.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:34.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:33.881+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:34.175 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:33.881+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:34.175 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:34.087+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:34.175 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:34.087+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:34.175 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:34.087+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:34.175 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:34.087+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:34.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:34.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:34.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:34.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:34.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:34.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:35.183 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:35.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:35.184 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:35.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:35.184 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:35.044+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:35.184 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:35.044+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:35.184 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:34.838+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:35.184 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:34.838+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:35.185 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:34.838+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:35.185 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:34.838+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:35.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:35.196+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:35.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:35.196+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:35.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:35.196+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:36.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:35.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:35.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:35.858+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:35.858+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:36.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:36.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:36.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:36.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:36.016+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:36.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:36.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:36.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:36.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:37.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:36.952+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5658 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:36.872+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:36.872+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:36.872+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:36.872+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:37.038+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:37.038+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:37.038+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:37.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:37.038+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:37.251+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:37.251+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:37.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:37.251+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:38.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:37.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:37.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:37.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:37.837+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:38.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:38.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:38.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:38.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:38.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:38.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:38.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:38.064+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:38.277+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:38.277+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:38.277+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:38.827+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:38.827+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:38.827+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:38.827+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:39.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:39.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:39.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:39.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:39.030+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:39.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:39.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:39.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:39.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:39.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:39.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:39.809+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:39.809+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:39.809+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:39.809+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:40.052+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:40.052+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:40.052+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:40.052+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:40.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:40.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:40.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:40.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:40.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:40.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:41.033 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:40.768+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:41.034 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:40.768+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:41.034 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:40.768+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:41.034 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:40.768+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:41.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:41.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:41.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:41.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:41.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:41.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:41.046+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:41.046+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:41.046+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:41.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:41.046+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:42.025 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:41.953+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5716 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:42.025 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:41.756+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:42.025 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:41.756+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:42.025 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:41.756+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:42.026 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:41.756+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:42.037+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:42.037+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:42.037+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:42.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:42.037+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:42.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:42.267+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:42.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:42.267+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:42.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:42.267+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:43.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:42.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:43.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:42.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:43.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:42.765+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:43.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:42.765+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:43.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:43.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:43.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:43.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:43.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:43.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:43.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:43.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:43.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:43.048+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:44.060 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:43.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:44.060 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:43.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:44.060 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:43.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:44.060 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:43.721+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:44.402 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:44.248+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:44.402 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:44.248+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:44.402 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:44.248+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:44.402 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:44.073+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:44.402 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:44.073+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:44.403 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:44.073+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:44.403 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:44.073+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:44.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:44.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:44.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:44.710+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:45.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:45.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:45.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:45.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:45.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:45.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:45.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:45.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:45.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:45.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:45.061+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:46.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:45.731+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:46.071 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:45.731+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:46.072 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:45.731+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:46.072 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:45.731+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:46.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:46.269+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:46.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:46.269+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:46.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:46.269+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:46.084+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:46.084+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:46.084+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:46.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:46.084+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:47.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:46.955+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5772 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:47.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:46.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:47.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:46.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:47.035 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:46.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:47.036 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:46.740+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:47.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:47.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:47.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:47.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:47.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:47.222+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:47.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:47.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:47.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:47.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:47.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:47.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:47.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:47.048+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:47.691+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:47.691+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:47.691+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:47.691+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:48.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:48.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:48.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:48.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:48.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:48.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:48.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:48.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:48.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:48.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:48.049+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:48.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:48.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:48.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:48.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:48.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:48.664+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:49.450 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:49.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:49.450 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:49.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:49.451 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:49.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:49.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.019+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:49.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.019+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:49.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.019+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:49.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.019+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:49.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:49.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:49.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:49.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:49.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:49.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:49.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:49.665+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:50.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:50.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:50.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:50.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:50.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:50.162+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:50.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.980+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:50.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.980+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:50.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.980+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:50.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:49.980+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:50.635+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:50.635+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:50.635+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:50.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:50.635+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:51.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:51.117+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:51.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:51.117+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:51.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:51.117+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:51.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:51.023+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:51.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:51.023+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:51.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:51.023+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:51.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:51.023+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:51.943 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:51.627+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:51.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:51.627+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:51.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:51.627+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:51.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:51.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:52.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:51.956+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5830 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:52.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:52.167+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:52.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:52.167+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:52.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:52.167+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:52.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:52.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:52.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:52.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:52.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:52.026+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:52.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:52.026+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:52.582+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:52.582+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:52.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:52.582+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:52.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:52.582+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:53.034+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:53.034+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:53.034+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:53.034+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:53.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:53.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:53.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:53.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:53.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:53.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:53.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:53.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:53.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:53.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:53.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:53.549+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:54.063+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:54.063+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:54.063+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:54.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:54.064+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:54.249+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:54.249+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:54.249+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:54.583+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:54.583+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:54.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:54.583+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:54.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:54.583+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:55.045+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:55.045+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:55.045+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:55.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:55.045+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:55.201+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:55.201+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:55.201+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:55.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:55.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:55.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:55.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:55.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:55.619+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:56.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:56.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:56.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:56.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:56.093+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:56.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:56.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:56.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:56.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:56.602+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:56.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:56.602+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:56.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:56.602+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:56.945 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:56.602+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:57.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:11:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:11:56.958+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5883 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:11:57.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:57.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:57.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:57.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:57.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:57.096+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:57.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:57.096+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:57.636 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:57.221+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:57.636 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:57.221+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:57.636 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:57.221+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:57.649+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:57.649+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:57.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:57.649+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:57.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:57.649+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:58.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:58.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:58.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:58.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:58.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:58.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:58.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:58.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:58.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:58.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:58.048+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:58.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:58.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:58.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:58.661+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:59.018+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:59.018+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:59.018+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:11:59.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:11:59.019+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:11:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:59.256+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:59.256+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:59.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:11:59.256+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:11:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:59.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:59.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:59.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:11:59.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:11:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:11:59.711+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.037+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.037+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.037+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:00.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.037+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:00.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:00.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:00.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:00.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:01.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:01.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:01.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:00.998+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:00.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:00.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:00.755+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:01.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:00.755+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:01.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:01.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:01.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:01.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:01.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:01.226+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:02.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:01.959+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5935 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:02.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:01.949+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:01.949+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:01.949+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:01.949+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:01.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:01.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:01.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:01.725+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:02.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:02.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:02.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:02.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:02.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:02.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:02.921+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:02.921+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:02.921+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:02.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:02.921+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:02.719+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:02.719+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:02.719+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:02.719+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:03.662 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:03.253+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:03.662 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:03.253+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:03.662 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:03.253+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:03.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:03.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:03.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:03.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:03.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:03.909+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:03.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:03.909+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:03.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:03.675+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:03.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:03.675+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:03.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:03.675+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:03.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:03.675+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:04.598 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:04.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:04.598 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:04.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:04.598 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:04.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:04.934 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:04.668+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:04.934 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:04.668+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:04.935 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:04.668+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:04.935 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:04.668+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:05.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:04.947+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:05.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:04.947+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:05.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:04.947+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:05.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:04.948+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:05.610 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:05.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:05.611 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:05.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:05.611 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:05.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:05.904 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:05.624+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:05.904 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:05.624+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:05.905 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:05.624+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:05.905 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:05.624+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:06.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:05.918+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:06.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:05.918+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:06.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:05.918+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:06.211 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:05.918+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:06.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:06.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:06.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:06.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:06.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:06.366+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:06.908 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:06.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:06.908 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:06.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:06.908 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:06.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:06.909 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:06.633+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:07.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:06.960+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 5982 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:07.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:06.921+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:07.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:06.921+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:07.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:06.921+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:07.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:06.921+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:07.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:07.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:07.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:07.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:07.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:07.326+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:07.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:07.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:07.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:07.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:07.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:07.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:07.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:07.656+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:08.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:07.965+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:08.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:07.965+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:08.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:07.965+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:08.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:07.965+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:08.660 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:08.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:08.661 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:08.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:08.661 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:08.364+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:08.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:08.674+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:08.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:08.674+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:08.674+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:08.674+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:09.340 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.007+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:09.340 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.007+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:09.340 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.007+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:09.340 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.007+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:09.615 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:09.353+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:09.615 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:09.353+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:09.615 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:09.353+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:09.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:09.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:09.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:09.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:10.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:10.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:10.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:10.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:09.970+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:10.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:10.398+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:10.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:10.398+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:10.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:10.398+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:10.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:10.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:10.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:10.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:10.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:10.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:10.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:10.646+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:11.350 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:11.004+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:11.350 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:11.004+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:11.350 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:11.004+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:11.350 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:11.004+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:11.681 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:11.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:11.681 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:11.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:11.681 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:11.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:11.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:11.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:11.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:11.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:11.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:11.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:11.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:11.694+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:12.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:11.961+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6031 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:12.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.042+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:12.664 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:12.401+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:12.665 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:12.401+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:12.665 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:12.401+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:12.678+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:12.678+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:12.678+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:12.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:12.678+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:13.393 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:13.393 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:13.394 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.998+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:13.394 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:12.998+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:13.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:13.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:13.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:13.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:13.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:13.407+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:13.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:13.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:13.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:13.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:13.634+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:14.405 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:14.033+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:14.405 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:14.033+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:14.406 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:14.033+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:14.406 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:14.033+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:14.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:14.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:14.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:14.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:14.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:14.418+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:14.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:14.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:14.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:14.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:14.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:14.683+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:15.378 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:15.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:15.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:15.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:15.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:15.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:15.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:15.042+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:15.663 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:15.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:15.664 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:15.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:15.664 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:15.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:15.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:15.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:15.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:15.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:15.676+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:16.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:16.035+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:16.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:16.035+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:16.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:16.035+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:16.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:16.035+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:16.670 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:16.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:16.670 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:16.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:16.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:16.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:16.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:16.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:16.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:16.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:16.949 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:16.683+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:16.950 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:16.683+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:17.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:16.961+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6081 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:17.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:17.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:17.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:17.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:17.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:17.048+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:17.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:17.048+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:17.693 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:17.453+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:17.693 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:17.453+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:17.694 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:17.453+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:17.705+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:17.705+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:17.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:17.705+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:17.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:17.705+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:18.430 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.010+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:18.430 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.010+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:18.430 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.010+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:18.431 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.010+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:18.699 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:18.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:18.700 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:18.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:18.700 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:18.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:18.712+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:18.712+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:18.712+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:18.712+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:19.398 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:19.398 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:19.398 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:19.399 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:18.982+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:19.659 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:19.411+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:19.659 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:19.411+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:19.659 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:19.411+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:19.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:19.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:19.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:19.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:19.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:19.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:19.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:19.672+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:20.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:19.964+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:20.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:19.964+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:20.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:19.964+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:20.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:19.964+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:20.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:20.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:20.644 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:20.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:20.644 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:20.385+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:20.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:20.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:20.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:20.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:20.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:20.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:20.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:20.656+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:21.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:20.960+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:21.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:20.960+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:21.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:20.960+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:21.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:20.960+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:21.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:21.412+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:21.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:21.412+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:21.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:21.412+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:21.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:21.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:21.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:21.648+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:22.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:21.962+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6136 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:21.924+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:21.924+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:21.924+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:22.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:21.924+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:22.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:22.408+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:22.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:22.408+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:22.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:22.408+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:22.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:22.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:22.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:22.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:22.619+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:23.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:22.971+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:23.380 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:22.971+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:23.380 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:22.971+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:23.380 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:22.971+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:23.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:23.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:23.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:23.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:23.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:23.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:23.607+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:23.607+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:23.607+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:23.607+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:24.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:23.941+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:24.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:23.941+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:24.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:23.941+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:24.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:23.941+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:24.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:24.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:24.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:24.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:24.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:24.379+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:24.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:24.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:24.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:24.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:24.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:24.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:24.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:24.572+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:25.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:24.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:25.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:24.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:24.970+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:25.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:24.970+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:25.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:25.343+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:25.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:25.343+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:25.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:25.343+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:25.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:25.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:25.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:25.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:25.579+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:26.343 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:25.987+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:26.343 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:25.987+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:26.343 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:25.987+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:26.344 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:25.987+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:26.597 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:26.355+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:26.597 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:26.355+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:26.597 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:26.355+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:26.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:26.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:26.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:26.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:26.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:26.610+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:26.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:26.610+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:27.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:26.963+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6190 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:26.965+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:26.965+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:26.965+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:27.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:26.965+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:27.615 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:27.354+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:27.616 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:27.354+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:27.616 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:27.354+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:27.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:27.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:27.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:27.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:27.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:28.390 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:28.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:28.391 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:28.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:28.391 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:28.000+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:28.391 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:28.000+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:28.648 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:28.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:28.648 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:28.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:28.649 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:28.403+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:28.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:28.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:28.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:28.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:28.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:28.661+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:29.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.031+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:29.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.031+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:29.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.031+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:29.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.031+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:29.675 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:29.398+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:29.676 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:29.398+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:29.676 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:29.398+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:29.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:29.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:29.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:29.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:29.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:29.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:29.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:29.688+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:30.378 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.991+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:30.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.991+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:30.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.991+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:30.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:29.991+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:30.645 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:30.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:30.645 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:30.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:30.645 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:30.391+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:30.658+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:30.658+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:30.658+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:30.658+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:31.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:31.002+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:31.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:31.002+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:31.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:31.002+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:31.401 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:31.002+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:31.663 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:31.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:31.663 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:31.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:31.663 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:31.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:31.952 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:31.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:31.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:31.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:31.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:31.676+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:31.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:31.676+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:32.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:31.964+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6248 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:32.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:32.038+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:32.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:32.038+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:32.038+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:32.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:32.038+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:32.697 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:32.423+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:32.698 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:32.423+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:32.698 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:32.423+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:32.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:32.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:32.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:32.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:32.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:32.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:32.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:32.710+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:33.418 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:33.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:33.418 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:33.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:33.419 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:33.061+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:33.419 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:33.061+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:33.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:33.431+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:33.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:33.431+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:33.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:33.431+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:34.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:33.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:34.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:33.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:34.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:33.732+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:34.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:33.732+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:34.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:34.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:34.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:34.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:34.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:34.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:34.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:34.093+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:34.672 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:34.427+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:34.673 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:34.427+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:34.673 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:34.427+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:34.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:34.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:34.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:34.686+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:35.393 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:35.122+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:35.393 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:35.122+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:35.393 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:35.122+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:35.394 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:35.123+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:35.702 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:35.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:35.702 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:35.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:35.703 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:35.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:35.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:35.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:35.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:35.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:35.715+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:36.416 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:36.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:36.417 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:36.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:36.417 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:36.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:36.417 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:36.171+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:36.677 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:36.429+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:36.678 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:36.429+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:36.678 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:36.429+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:36.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:36.690+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:36.954 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:36.690+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:36.954 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:36.690+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:36.954 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:36.691+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:37.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:36.967+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6305 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:37.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:37.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:37.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:37.159+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:37.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:37.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:37.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:37.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:37.645+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:37.645+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:37.645+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:37.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:37.645+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:38.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:38.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:38.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:38.189+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:38.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:38.499+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:38.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:38.499+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:38.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:38.499+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:38.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:38.637+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:38.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:38.637+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:38.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:38.637+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:38.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:38.637+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:39.536 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:39.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:39.536 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:39.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:39.536 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:39.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:39.537 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:39.225+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:39.943 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:39.549+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:39.943 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:39.549+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:39.943 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:39.549+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:39.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:39.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:39.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:39.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:39.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:39.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:39.944 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:39.680+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:40.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:40.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:40.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:40.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:40.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:40.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:40.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:40.274+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:40.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:40.588+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:40.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:40.588+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:40.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:40.588+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:40.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:40.645+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:40.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:40.645+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:40.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:40.645+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:40.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:40.645+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:41.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:41.235+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:41.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:41.235+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:41.584 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:41.235+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:41.584 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:41.235+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:41.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:41.596+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:41.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:41.596+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:41.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:41.596+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:41.955 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:41.659+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:41.955 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:41.659+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:41.659+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:41.659+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:42.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:41.968+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6364 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:42.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:42.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:42.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:42.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:42.193+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:42.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:42.194+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:42.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:42.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:42.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:42.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:42.602+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:42.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:42.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:42.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:42.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:42.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:42.656+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:42.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:42.656+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:43.192+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:43.192+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:43.192+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:43.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:43.193+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:43.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:43.646+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:43.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:43.646+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:43.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:43.646+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:43.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:43.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:43.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:43.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:43.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:43.701+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:43.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:43.701+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:44.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:44.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:44.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:44.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:44.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:44.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:44.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:44.229+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:44.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:44.692+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:44.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:44.692+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:44.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:44.692+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:44.657+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:44.657+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:44.657+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:44.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:44.657+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:45.599 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:45.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:45.599 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:45.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:45.599 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:45.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:45.599 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:45.274+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:45.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:45.675+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:45.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:45.675+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:45.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:45.675+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:45.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:45.612+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:45.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:45.612+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:45.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:45.612+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:45.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:45.612+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:46.588 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:46.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:46.588 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:46.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:46.589 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:46.275+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:46.589 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:46.275+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:46.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:46.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:46.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:46.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:46.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:46.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:46.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:46.602+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:46.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:46.602+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:46.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:46.602+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:46.602+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:47.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:46.969+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6421 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:47.622 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:47.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:47.623 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:47.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:47.623 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:47.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:47.623 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:47.261+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:47.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:47.636+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:47.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:47.636+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:47.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:47.636+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:47.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:47.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:47.646+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:47.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:47.647+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:48.597 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:48.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:48.597 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:48.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:48.598 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:48.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:48.598 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:48.262+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:48.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:48.685+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:48.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:48.685+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:48.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:48.685+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:48.611+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:48.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:48.611+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:48.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:48.611+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:48.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:48.611+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:49.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:49.284+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:49.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:49.284+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:49.575 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:49.284+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:49.576 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:49.285+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:49.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:49.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:49.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:49.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:49.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:49.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:49.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:49.589+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:49.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:49.589+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:49.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:49.589+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:49.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:49.589+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:50.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:50.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:50.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:50.240+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:50.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:50.731+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:50.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:50.731+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:50.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:50.731+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:50.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:50.561+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:50.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:50.561+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:50.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:50.561+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:50.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:50.561+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:51.552 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:51.288+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:51.553 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:51.288+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:51.553 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:51.288+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:51.553 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:51.288+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:51.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:51.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:51.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:51.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:51.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:51.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:51.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:51.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:51.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:51.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:51.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:51.566+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:51.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:51.566+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:52.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:51.970+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6480 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:52.600 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:52.266+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:52.601 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:52.266+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:52.601 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:52.266+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:52.601 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:52.266+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:52.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:52.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:52.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:52.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:52.614+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:52.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:52.746+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:52.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:52.746+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:52.961 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:52.746+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:53.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:53.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:53.220+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:53.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:53.220+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:53.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:53.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:53.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:53.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:53.585+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:53.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:53.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:53.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:53.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:53.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:53.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:54.533 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:54.234+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:54.533 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:54.234+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:54.533 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:54.234+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:54.533 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:54.234+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:54.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:54.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:54.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:54.546+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:54.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:54.769+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:54.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:54.769+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:54.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:54.769+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:55.276+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:55.276+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:55.276+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:55.276+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:55.773 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:55.533+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:55.773 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:55.533+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:55.773 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:55.533+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:55.773 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:55.533+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:56.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:55.786+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:56.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:55.786+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:56.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:55.786+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:56.471 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:56.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:56.471 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:56.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:56.471 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:56.229+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:56.471 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:56.229+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:56.725 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:56.484+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:56.725 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:56.484+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:56.725 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:56.484+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:56.725 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:56.484+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:57.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:12:56.971+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6537 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:12:57.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:56.738+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:57.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:56.738+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:56.738+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:57.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:57.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:57.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:57.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:57.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:57.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:57.521 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:57.250+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:57.533+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:57.533+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:57.533+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:57.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:57.534+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:57.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:57.717+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:57.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:57.717+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:57.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:57.717+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:58.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:58.286+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:58.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:58.286+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:58.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:58.286+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:58.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:58.286+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:58.543+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:58.543+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:58.543+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:58.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:58.543+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:58.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:58.750+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:58.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:58.750+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:58.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:58.750+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:59.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:59.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:59.531 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:59.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:59.531 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:59.263+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:12:59.531 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:12:59.263+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:12:59.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:59.717+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:59.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:59.717+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:59.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:12:59.717+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:12:59.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:59.543+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:59.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:59.543+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:59.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:59.543+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:12:59.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:12:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:12:59.543+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:00.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:00.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:00.300+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:00.301+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:00.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:00.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:00.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:00.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:00.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:00.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:00.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:00.527+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:00.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:00.527+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:00.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:00.527+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:00.961 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:00.527+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:01.308+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:01.308+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:01.308+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:01.308+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:01.539+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:01.539+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:01.539+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:01.539+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:02.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:01.972+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6596 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:02.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:01.726+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:02.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:01.726+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:02.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:01.726+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:02.328+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:02.328+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:02.328+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:02.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:02.328+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:02.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:02.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:02.503+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:02.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:02.503+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:03.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:02.759+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:03.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:02.759+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:03.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:02.759+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:03.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:03.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:03.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:03.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:03.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:03.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:03.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:03.367+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:03.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:03.466+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:03.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:03.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:03.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:03.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:03.710 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:03.467+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:04.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:03.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:04.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:03.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:04.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:03.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:04.392+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:04.392+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:04.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:04.392+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:04.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:04.392+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:04.511+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:04.511+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:04.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:04.511+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:04.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:04.511+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:05.189 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:04.749+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:05.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:04.749+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:05.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:04.749+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:05.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:05.406+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:05.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:05.406+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:05.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:05.406+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:05.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:05.406+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:05.744 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:05.483+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:05.744 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:05.483+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:05.744 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:05.483+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:05.744 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:05.483+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:06.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:05.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:06.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:05.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:06.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:05.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:06.440+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:06.440+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:06.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:06.440+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:06.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:06.440+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:06.487+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:06.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:06.487+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:06.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:06.487+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:06.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:06.487+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:07.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:06.972+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6653 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:07.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:06.798+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:07.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:06.798+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:07.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:06.798+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:07.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:07.411+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:07.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:07.411+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:07.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:07.411+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:07.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:07.411+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:07.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:07.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:07.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:07.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:07.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:07.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:07.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:07.530+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:08.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:07.812+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:08.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:07.812+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:08.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:07.812+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:08.397+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:08.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:08.397+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:08.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:08.397+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:08.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:08.397+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:08.571+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:08.571+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:08.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:08.571+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:08.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:08.571+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:09.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:08.826+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:09.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:08.826+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:09.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:08.826+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:09.431+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:09.431+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:09.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:09.431+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:09.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:09.431+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:09.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:09.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:09.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:09.546+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:09.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:09.546+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:10.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:09.785+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:10.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:09.785+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:10.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:09.785+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:10.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:10.467+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:10.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:10.467+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:10.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:10.467+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:10.530 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:10.467+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:10.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:10.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:10.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:10.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:10.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:10.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:10.542+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:10.542+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:10.542+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:10.542+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:11.790 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:11.508+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:11.508+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:11.508+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:11.508+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:11.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:11.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:11.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:11.791 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:11.587+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:12.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:11.973+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6712 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:12.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:11.803+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:12.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:11.803+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:12.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:11.803+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:12.812 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:12.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:12.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:12.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:12.638+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:12.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:12.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:12.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:12.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:12.504+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:13.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:12.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:13.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:12.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:13.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:12.825+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:13.792 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:13.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:13.793 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:13.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:13.793 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:13.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:13.793 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:13.518+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:13.793 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:13.601+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:13.793 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:13.601+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:13.794 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:13.601+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:13.794 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:13.601+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:14.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:13.805+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:14.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:13.805+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:14.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:13.805+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:14.825 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:14.562+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:14.826 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:14.562+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:14.826 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:14.562+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:14.826 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:14.562+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:14.826 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:14.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:14.826 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:14.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:14.826 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:14.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:14.827 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:14.536+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:15.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:14.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:15.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:14.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:15.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:14.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:15.561 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:15.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:15.562 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:15.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:15.562 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:15.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:15.562 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:15.509+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:15.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:15.822+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:15.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:15.822+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:15.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:15.822+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:15.574+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:15.574+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:15.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:15.574+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:15.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:15.574+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:16.596 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:16.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:16.597 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:16.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:16.597 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:16.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:16.597 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:16.490+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:16.609+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:16.609+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:16.609+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:16.609+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:16.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:16.795+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:16.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:16.795+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:16.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:16.795+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:17.447 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:16.975+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6769 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:17.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:17.644+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:17.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:17.644+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:17.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:17.644+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:17.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:17.644+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:17.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:17.460+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:17.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:17.460+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:17.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:17.460+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:17.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:17.460+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:18.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:17.835+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:18.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:17.835+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:18.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:17.835+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:18.827 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:18.499+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:18.827 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:18.499+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:18.827 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:18.499+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:18.827 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:18.499+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:18.828 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:18.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:18.828 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:18.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:18.828 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:18.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:18.828 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:18.686+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:18.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:18.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:18.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:19.637 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:19.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:19.638 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:19.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:19.638 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:19.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:19.638 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:19.471+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:19.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:19.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:19.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:19.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:19.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:19.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:19.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:19.650+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:19.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:19.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:19.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:19.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:19.959 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:19.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:20.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:20.477+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:20.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:20.477+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:20.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:20.477+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:20.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:20.477+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:20.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:20.685+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:20.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:20.685+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:20.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:20.685+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:20.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:20.685+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:20.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:20.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:20.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:20.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:20.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:20.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:21.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:21.669+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:21.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:21.669+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:21.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:21.669+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:21.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:21.669+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:21.465+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:21.465+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:21.465+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:21.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:21.465+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:22.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:21.976+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6828 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:21.907+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:21.907+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:21.907+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:22.697 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:22.512+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:22.702 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:22.512+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:22.702 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:22.512+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:22.702 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:22.512+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:22.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:22.887+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:22.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:22.887+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:22.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:22.887+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:22.709+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:22.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:22.709+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:22.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:22.709+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:22.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:22.709+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:23.874 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:23.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:23.874 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:23.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:23.874 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:23.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:23.874 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:23.491+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:23.874 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:23.667+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:23.875 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:23.667+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:23.875 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:23.667+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:23.875 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:23.667+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:24.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:23.887+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:24.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:23.887+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:24.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:23.887+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:24.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:24.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:24.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:24.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:24.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:24.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:24.914 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:24.496+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:24.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:24.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:24.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:24.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:24.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:24.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:24.915 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:24.642+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:25.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:24.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:25.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:24.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:25.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:24.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:25.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:25.463+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:25.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:25.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:25.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:25.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:25.660 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:25.464+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:25.917 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:25.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:25.917 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:25.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:25.917 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:25.672+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:25.917 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:25.672+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:26.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:25.930+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:26.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:25.930+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:26.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:25.930+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:26.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:26.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:26.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:26.445+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:26.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:26.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:26.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:26.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:26.681+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:27.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:26.978+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6885 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:27.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:26.967+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:27.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:26.967+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:27.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:26.967+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:27.922 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:27.479+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:27.923 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:27.479+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:27.923 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:27.479+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:27.923 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:27.479+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:27.923 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:27.718+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:27.923 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:27.718+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:27.923 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:27.718+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:27.924 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:27.718+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:28.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:27.936+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:28.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:27.936+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:28.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:27.936+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:28.430+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:28.430+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:28.430+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:28.431+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:28.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:28.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:28.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:28.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:28.670+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:29.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:28.886+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:29.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:28.886+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:29.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:28.886+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:29.689 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:29.422+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:29.690 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:29.422+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:29.690 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:29.422+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:29.690 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:29.422+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:29.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:29.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:29.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:29.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:29.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:29.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:29.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:29.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:29.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:29.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:29.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:29.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:29.703+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:30.417+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:30.417+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:30.417+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:30.418+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:30.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:30.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:30.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:31.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:30.743+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:31.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:30.743+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:31.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:30.743+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:31.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:30.744+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:31.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:31.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:31.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:31.444+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:32.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:31.979+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 6944 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:32.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:31.876+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:32.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:31.876+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:32.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:31.876+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:32.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:31.770+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:32.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:31.770+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:32.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:31.770+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:32.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:31.770+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:32.413+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:32.413+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:32.413+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:32.413+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:33.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:32.882+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:33.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:32.882+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:33.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:32.882+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:33.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:32.793+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:33.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:32.793+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:33.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:32.793+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:33.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:32.793+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:33.418+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:33.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:33.418+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:33.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:33.418+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:33.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:33.418+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:34.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:33.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:34.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:33.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:34.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:33.849+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:33.818+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:33.818+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:33.818+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:33.818+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:34.436+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:34.436+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:34.436+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:34.436+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:35.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:34.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:35.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:34.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:35.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:34.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:34.782+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:35.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:34.782+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:35.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:34.782+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:35.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:34.782+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:35.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:35.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:35.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:35.396+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:36.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:35.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:36.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:35.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:36.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:35.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:35.805+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:35.805+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:35.805+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:35.805+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:36.369+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:36.369+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:36.369+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:36.369+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:37.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:36.981+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7001 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:37.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:36.889+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:36.889+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:37.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:36.889+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:36.811+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:36.811+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:36.811+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:36.811+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:37.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:37.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:37.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:37.352+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:38.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:37.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:38.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:37.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:38.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:37.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:37.841+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:37.841+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:37.841+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:38.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:37.841+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:38.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:38.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:38.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:38.352+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:38.889+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:38.889+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:38.889+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:38.853+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:39.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:38.853+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:39.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:38.853+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:39.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:38.853+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:39.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:39.374+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:39.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:39.374+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:39.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:39.374+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:39.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:39.374+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:39.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:39.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:39.872+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:39.835+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:39.835+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:39.835+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:39.835+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:40.333+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:40.333+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:40.333+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:40.334+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:41.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:40.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:41.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:40.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:41.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:40.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:41.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:40.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:41.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:40.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:41.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:40.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:41.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:40.874+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:41.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:41.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:41.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:41.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:41.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:41.301+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:41.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:41.301+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:42.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:41.982+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7060 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:42.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:41.865+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:42.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:41.865+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:42.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:41.865+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:42.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:41.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:42.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:41.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:42.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:41.874+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:42.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:41.874+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:42.285+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:42.285+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:42.285+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:42.285+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:43.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:42.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:43.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:42.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:43.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:42.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:43.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:42.887+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:43.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:42.887+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:43.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:42.887+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:43.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:42.887+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:43.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:43.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:43.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:43.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:43.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:43.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:43.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:43.319+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:44.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:43.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:44.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:43.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:44.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:43.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:43.920+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:44.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:43.920+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:44.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:43.920+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:44.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:43.920+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:44.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:44.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:44.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:44.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:44.292+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:44.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:44.292+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:45.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:44.929+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:45.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:44.929+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:45.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:44.929+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:45.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:44.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:45.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:44.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:45.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:44.886+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:45.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:44.886+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:45.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:45.299+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:45.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:45.299+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:45.583 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:45.299+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:45.584 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:45.300+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:46.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:45.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:46.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:45.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:46.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:45.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:46.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:45.865+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:46.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:45.865+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:46.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:45.865+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:46.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:45.865+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:46.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:46.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:46.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:46.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:46.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:46.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:46.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:46.273+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:47.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:46.983+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7117 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:46.828+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:46.828+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:46.828+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:47.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:46.828+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:46.932+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:46.932+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:47.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:46.932+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:47.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:47.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:47.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:47.272+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:47.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:47.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:47.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:47.869+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:48.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:47.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:48.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:47.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:48.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:47.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:48.296+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:48.296+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:48.296+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:48.296+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:49.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:48.915+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:49.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:48.915+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:49.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:48.915+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:49.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:48.915+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:49.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:48.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:49.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:48.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:49.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:48.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:49.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:49.327+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:49.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:49.327+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:49.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:49.327+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:49.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:49.327+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:50.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:49.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:50.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:49.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:50.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:49.897+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:50.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:49.897+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:50.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:49.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:50.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:49.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:50.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:49.917+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:50.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:50.302+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:50.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:50.302+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:50.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:50.302+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:50.632 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:50.302+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:50.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:50.863+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:50.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:50.863+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:50.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:50.863+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:50.953 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:50.863+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:51.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:50.966+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:51.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:50.966+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:51.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:50.966+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:51.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:51.347+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:51.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:51.347+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:51.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:51.347+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:51.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:51.347+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:52.209 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:51.984+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7176 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:51.846+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:51.846+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:51.846+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:52.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:51.846+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:52.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:51.946+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:52.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:51.946+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:52.211 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:51.946+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:52.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:52.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:52.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:52.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:52.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:52.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:52.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:52.350+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:53.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:52.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:53.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:52.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:53.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:52.941+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:53.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:52.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:53.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:52.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:53.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:52.888+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:53.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:52.888+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:53.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:53.375+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:53.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:53.375+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:53.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:53.375+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:53.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:53.375+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:54.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:53.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:54.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:53.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:54.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:53.894+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:54.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:53.894+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:54.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:53.952+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:54.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:53.952+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:54.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:53.952+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:54.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:54.389+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:54.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:54.389+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:54.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:54.389+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:54.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:54.389+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:55.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:54.930+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:55.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:54.930+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:55.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:54.930+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:55.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:54.877+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:55.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:54.877+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:55.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:54.877+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:55.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:54.877+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:55.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:55.341+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:55.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:55.341+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:55.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:55.341+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:55.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:55.341+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:55.879+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:55.879+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:55.879+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:55.879+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:55.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:55.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:55.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:55.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:55.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:55.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:56.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:56.375+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:56.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:56.375+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:56.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:56.375+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:56.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:56.375+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:57.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:13:56.985+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7233 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:13:57.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:56.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:57.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:56.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:57.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:56.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:57.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:56.928+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:56.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:56.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:57.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:56.919+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:57.326+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:57.326+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:57.326+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:57.326+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:58.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:57.894+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:58.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:57.894+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:58.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:57.894+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:58.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:57.937+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:58.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:57.937+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:58.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:57.937+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:58.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:57.937+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:58.298+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:58.298+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:58.298+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:58.298+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:59.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:58.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:59.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:58.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:59.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:58.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:13:59.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:58.973+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:13:59.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:58.912+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:59.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:58.912+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:59.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:58.912+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:13:59.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:59.322+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:59.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:59.322+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:59.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:59.322+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:13:59.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:13:59.322+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:00.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:59.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:00.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:59.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:00.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:13:59.927+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:00.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:59.926+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:00.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:59.926+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:00.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:59.926+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:00.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:13:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:13:59.926+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:00.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:00.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:00.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:00.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:00.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:00.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:00.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:00.335+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:01.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:00.898+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:01.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:00.898+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:01.210 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:00.898+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:01.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:00.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:01.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:00.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:01.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:00.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:01.211 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:00.973+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:01.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:01.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:01.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:01.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:01.350+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:02.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:01.986+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7292 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:02.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:01.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:02.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:01.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:02.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:01.880+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:01.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:01.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:01.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:02.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:01.945+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:02.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:02.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:02.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:02.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:02.309+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:03.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:02.904+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:03.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:02.904+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:03.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:02.904+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:03.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:02.967+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:03.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:02.967+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:03.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:02.967+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:03.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:02.967+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:03.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:03.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:03.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:03.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:03.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:03.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:03.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:03.297+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:04.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:03.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:04.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:03.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:04.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:03.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:04.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:03.960+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:04.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:03.960+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:04.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:03.960+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:04.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:03.960+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:04.298+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:04.298+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:04.298+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:04.298+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:05.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:04.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:05.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:04.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:05.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:04.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:05.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:04.926+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:05.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:04.926+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:05.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:04.926+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:05.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:04.926+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:05.339+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:05.339+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:05.339+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:05.339+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:06.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:05.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:06.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:05.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:06.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:05.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:06.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:05.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:06.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:05.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:06.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:05.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:06.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:05.945+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:06.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:06.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:06.315+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:06.315+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:07.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:06.987+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7348 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:07.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:06.860+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:07.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:06.860+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:07.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:06.860+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:07.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:06.992+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:07.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:06.992+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:07.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:06.992+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:07.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:06.992+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:07.329+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:07.329+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:07.329+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:07.329+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:08.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:07.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:08.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:07.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:08.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:07.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:08.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:07.997+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:08.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:07.997+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:08.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:07.997+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:08.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:07.997+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:08.308+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:08.308+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:08.308+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:08.308+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:09.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:08.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:09.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:08.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:09.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:08.888+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:09.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:09.035+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:09.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:09.035+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:09.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:09.035+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:09.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:09.035+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:09.320+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:09.320+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:09.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:09.320+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:09.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:09.320+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:10.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:09.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:10.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:09.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:10.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:09.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:10.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:10.024+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:10.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:10.024+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:10.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:10.024+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:10.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:10.024+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:10.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:10.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:10.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:10.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:10.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:10.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:10.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:10.351+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:11.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:10.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:11.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:10.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:11.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:10.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:11.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:11.056+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:11.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:11.056+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:11.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:11.056+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:11.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:11.056+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:11.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:11.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:11.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:11.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:11.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:11.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:11.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:11.378+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:12.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:11.988+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7406 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:12.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:11.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:12.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:11.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:12.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:11.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:12.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:12.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:12.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:12.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:12.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:12.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:12.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:12.062+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:12.371+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:12.371+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:12.371+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:12.371+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:13.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:12.876+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:13.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:12.876+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:13.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:12.876+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:13.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:13.034+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:13.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:13.034+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:13.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:13.034+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:13.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:13.034+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:13.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:13.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:13.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:13.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:13.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:13.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:13.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:13.378+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:14.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:13.845+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:14.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:13.845+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:14.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:13.845+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:14.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:14.068+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:14.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:14.068+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:14.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:14.068+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:14.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:14.068+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:14.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:14.336+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:14.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:14.336+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:14.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:14.336+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:14.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:14.336+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:15.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:14.871+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:15.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:14.871+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:15.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:14.871+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:15.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:15.048+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:15.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:15.048+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:15.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:15.048+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:15.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:15.048+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:15.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:15.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:15.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:15.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:15.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:15.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:15.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:15.358+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:16.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:15.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:16.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:15.824+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:16.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:15.824+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:16.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:16.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:16.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:16.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:16.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:16.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:16.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:16.005+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:16.337+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:16.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:16.337+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:16.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:16.337+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:16.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:16.337+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:17.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:16.991+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7463 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:17.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:16.797+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:17.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:16.797+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:17.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:16.797+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:17.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:17.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:17.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:17.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.043+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:17.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:17.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:17.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:17.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:17.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:17.313+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:17.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:17.314+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:18.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.999+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:18.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.999+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:18.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.999+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:18.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:17.999+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:18.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:17.750+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:18.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:17.750+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:18.209 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:17.750+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:18.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:18.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:18.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:18.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:18.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:18.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:18.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:18.330+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:18.790+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:18.790+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:18.790+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:19.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:18.953+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:19.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:18.953+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:19.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:18.953+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:19.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:18.953+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:19.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:19.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:19.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:19.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:19.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:19.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:19.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:19.309+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:19.976 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:19.819+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:19.976 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:19.819+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:19.976 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:19.819+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:20.292 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:19.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:20.292 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:19.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:20.292 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:19.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:20.292 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:19.990+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:20.306+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:20.306+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:20.306+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:20.306+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:21.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:20.788+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:21.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:20.788+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:21.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:20.788+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:21.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:21.008+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:21.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:21.008+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:21.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:21.008+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:21.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:21.008+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:21.306+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:21.306+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:21.306+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:21.307+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:22.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:21.992+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7522 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:21.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:21.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:21.823+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:22.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:22.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:22.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:22.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.013+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:22.615 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:22.273+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:22.615 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:22.273+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:22.615 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:22.273+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:22.616 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:22.273+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:23.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:22.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:23.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:22.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:23.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:22.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:23.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:23.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:23.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:23.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:22.989+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:23.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:23.226+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:23.226+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:23.226+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:23.226+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:24.176 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:23.835+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:24.236 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:23.835+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:24.236 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:23.835+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:24.236 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:23.994+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:24.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:23.994+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:24.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:23.994+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:24.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:23.994+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:24.190+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:24.190+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:24.190+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:24.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:24.190+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:25.164 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:24.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:25.164 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:24.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:25.165 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:24.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:25.165 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:24.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:25.165 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:24.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:25.165 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:24.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:25.166 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:24.980+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:25.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:25.178+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:25.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:25.178+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:25.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:25.178+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:25.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:25.178+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:26.157 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:25.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:26.157 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:25.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:26.157 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:25.878+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:26.157 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.021+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:26.157 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.021+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:26.158 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.021+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:26.158 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.021+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:26.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:26.170+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:26.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:26.170+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:26.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:26.170+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:26.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:26.170+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:27.167 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:26.993+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7578 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:26.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:26.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:26.862+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.986+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.986+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.986+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:27.168 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:26.986+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:27.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:27.181+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:27.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:27.181+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:27.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:27.181+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:27.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:27.181+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:28.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:27.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:28.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:27.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:28.190 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:27.885+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:28.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:27.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:28.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:27.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:28.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:27.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:28.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:27.998+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:28.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:28.204+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:28.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:28.204+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:28.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:28.204+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:28.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:28.204+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:29.033 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:28.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:29.034 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:28.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:29.034 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:28.838+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:29.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:29.204+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:29.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:29.204+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:29.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:29.204+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:29.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:29.204+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:29.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:29.047+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:29.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:29.047+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:29.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:29.047+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:29.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:29.047+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:30.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:29.836+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:30.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:29.836+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:30.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:29.836+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:30.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:30.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:30.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:30.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.042+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:30.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:30.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:30.230+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:30.231+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:30.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:30.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:31.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:30.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:31.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.995+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:31.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.995+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:31.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.995+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:31.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:30.995+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:31.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:31.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:31.236+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:31.236+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:32.196 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:31.994+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7637 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:32.196 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:31.873+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:32.197 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:31.873+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:32.197 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:31.873+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:32.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:32.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:32.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:32.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:32.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:32.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:32.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:32.010+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:32.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:32.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:32.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:32.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:32.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:32.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:32.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:32.210+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:33.202 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:32.900+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:33.203 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:32.900+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:33.203 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:32.900+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:33.203 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:33.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:33.203 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:33.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:33.203 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:33.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:33.204 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:33.025+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:33.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:33.216+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:33.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:33.216+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:33.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:33.216+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:33.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:33.216+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:34.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:33.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:34.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:33.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:34.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:33.913+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:34.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:34.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:34.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:34.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:34.031+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:34.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:34.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:34.222+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:34.222+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:35.175 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:34.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:35.175 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:34.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:35.175 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:34.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:35.176 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:35.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:35.176 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:35.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:35.176 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:35.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:35.176 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:35.052+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:35.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:35.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:35.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:35.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:35.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:35.189+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:35.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:35.189+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:36.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:36.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:36.159+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:36.159+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:35.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:35.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:35.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:36.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:36.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:36.062+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:36.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:36.062+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:37.207 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:36.996+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7694 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:37.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:36.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:37.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:36.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:37.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:36.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:37.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:37.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:37.205+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:37.205+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:37.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:37.032+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:37.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:37.032+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:37.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:37.032+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:37.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:37.032+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:38.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:38.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:38.225+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:38.225+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:38.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:37.977+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:37.977+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:38.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:37.977+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:38.067+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:38.067+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:38.067+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:38.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:38.067+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:39.212+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:39.212+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:39.212+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:39.212+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:39.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:38.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:39.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:38.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:39.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:38.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:39.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:39.103+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:39.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:39.103+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:39.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:39.103+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:39.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:39.103+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:40.233 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:39.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:40.233 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:39.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:40.233 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:39.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:40.233 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:40.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:40.234 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:40.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:40.234 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:40.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:40.234 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:40.145+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:40.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:40.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:40.247+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:40.247+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:41.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:41.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:41.240+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:41.240+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:41.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:41.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:41.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:41.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:41.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:41.008+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:41.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:41.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:41.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:41.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:41.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:41.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:41.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:41.181+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:42.203 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:41.997+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7753 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:42.204 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:42.051+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:42.204 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:42.051+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:42.204 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:42.051+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:42.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:42.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:42.272+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:42.272+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:42.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:42.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:42.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:42.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:42.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:42.217+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:42.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:42.217+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:43.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:43.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:43.237+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:43.238+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:43.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:43.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:43.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:43.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:43.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:43.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:43.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:43.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:43.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:43.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:43.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:43.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:43.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:43.233+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:44.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:44.283+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:44.283+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:44.283+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:44.459 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:44.283+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:44.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:44.035+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:44.459 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:44.035+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:44.460 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:44.035+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:44.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:44.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:44.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:44.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:44.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:44.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:44.460 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:44.200+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:45.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:45.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:45.278+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:45.278+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:45.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:45.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:45.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:45.202+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:45.202+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:45.202+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:45.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:45.202+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:46.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:46.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:46.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:46.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:46.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:46.265+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:46.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:46.265+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:46.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:46.028+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:46.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:46.028+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:46.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:46.028+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:46.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:46.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:46.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:46.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:46.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:46.191+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:46.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:46.192+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:47.275 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:46.997+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7810 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:47.276 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:47.005+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:47.276 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:47.005+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:47.276 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:47.005+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:47.276 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:47.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:47.277 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:47.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:47.277 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:47.236+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:47.277 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:47.236+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:47.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:47.289+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:47.289+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:47.289+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:47.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:47.289+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:48.260 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:48.016+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:48.261 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:48.016+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:48.261 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:48.016+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:48.261 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:48.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:48.261 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:48.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:48.261 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:48.254+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:48.261 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:48.254+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:48.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:48.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:48.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:48.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:48.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:48.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:48.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:48.273+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:49.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:49.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:49.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:49.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:49.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:49.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:49.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:49.294+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:49.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:49.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:49.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:49.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:49.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:49.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:49.232+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:49.232+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:49.232+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:49.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:49.232+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:50.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:50.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:50.250+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:50.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:50.250+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:50.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:50.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:50.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:50.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:50.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:50.038+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:50.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:50.252+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:50.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:50.252+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:50.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:50.252+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:50.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:50.252+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:51.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:51.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:51.269+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:51.269+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:51.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:51.073+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:51.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:51.073+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:51.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:51.073+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:51.244+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:51.244+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:51.244+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:51.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:51.244+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:52.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:51.999+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7869 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:52.228+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:52.228+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:52.228+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:52.228+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:52.083+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:52.083+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:52.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:52.083+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:52.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:52.228+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:52.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:52.228+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:52.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:52.228+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:52.459 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:52.228+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:53.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:53.181+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:53.181+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:53.181+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:53.181+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:53.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:53.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:53.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:53.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:53.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:53.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:53.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:53.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:53.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:53.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:53.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:53.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:53.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:53.209+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:54.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:54.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:54.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:54.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:54.219+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:54.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:54.163+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:54.163+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:54.163+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:54.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:54.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:54.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:54.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:54.233+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:55.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:55.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:55.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:55.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:55.238+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:55.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:55.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:55.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:55.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:55.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:55.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:55.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:55.220+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:55.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:55.220+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:55.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:55.220+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:55.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:55.220+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:56.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:56.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:56.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:56.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:56.279+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:56.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:56.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:56.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:56.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:56.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:56.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:56.207+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:56.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:56.207+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:56.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:56.207+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:56.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:56.207+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:57.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:14:57.000+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7926 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:14:57.253 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:57.239+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:57.253 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:57.239+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:57.253 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:57.239+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:57.253 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:57.239+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:57.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:57.283+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:57.283+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:57.283+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:57.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:57.283+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:57.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:57.265+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:57.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:57.265+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:57.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:57.265+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:58.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:58.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:58.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:58.297+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:58.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:58.297+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:58.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:58.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:58.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:58.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:58.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:58.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:58.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:58.273+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:58.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:58.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:58.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:58.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:58.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:58.272+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:59.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:59.246+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:59.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:59.246+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:59.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:14:59.246+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:14:59.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:59.318+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:59.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:59.318+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:59.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:59.318+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:14:59.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:14:59.318+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:14:59.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:59.280+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:59.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:59.280+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:59.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:59.280+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:14:59.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:14:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:14:59.280+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:00.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:00.285+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:00.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:00.285+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:00.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:00.285+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:00.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:00.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:00.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:00.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:00.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:00.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:00.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:00.330+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:00.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:00.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:00.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:00.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:00.267+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:01.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:01.291+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:01.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:01.291+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:01.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:01.291+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:01.334+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:01.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:01.334+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:01.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:01.334+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:01.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:01.334+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:01.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:01.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:01.254+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:01.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:01.254+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:02.285 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:02.001+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 7985 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:02.285 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:02.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:02.285 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:02.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:02.285 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:02.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:02.285 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:02.209+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:02.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:02.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:02.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:02.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:02.381+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:02.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:02.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:02.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:02.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:02.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:02.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:03.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:03.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:03.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:03.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:03.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:03.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:03.399+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:03.180+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:03.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:03.180+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:03.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:03.180+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:03.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:03.180+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:03.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:03.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:03.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:03.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:03.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:03.276+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:04.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:04.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:04.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:04.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:04.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:04.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:04.380+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:04.134+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:04.134+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:04.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:04.134+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:04.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:04.134+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:04.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:04.297+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:04.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:04.297+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:04.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:04.297+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:05.388 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:05.312+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:05.389 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:05.312+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:05.389 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:05.312+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:05.389 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:05.127+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:05.389 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:05.127+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:05.389 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:05.127+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:05.390 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:05.127+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:05.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:05.402+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:05.402+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:05.402+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:05.402+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:06.408 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:06.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:06.408 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:06.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:06.408 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:06.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:06.409 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:06.116+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:06.409 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:06.268+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:06.409 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:06.268+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:06.409 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:06.268+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:06.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:06.421+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:06.421+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:06.421+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:06.421+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:07.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:07.003+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8042 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:07.247 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:07.150+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:07.247 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:07.150+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:07.247 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:07.150+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:07.247 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:07.150+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:07.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:07.405+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:07.405+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:07.405+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:07.405+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:07.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:07.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:07.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:07.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:07.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:07.260+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:08.400 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:08.296+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:08.401 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:08.296+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:08.401 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:08.296+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:08.401 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:08.149+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:08.401 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:08.149+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:08.401 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:08.149+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:08.402 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:08.149+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:08.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:08.414+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:08.414+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:08.414+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:08.414+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:09.445 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:09.315+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:09.446 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:09.315+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:09.446 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:09.315+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:09.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:09.178+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:09.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:09.178+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:09.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:09.178+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:09.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:09.178+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:09.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:09.456+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:09.456+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:09.456+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:09.456+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:10.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:10.362+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:10.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:10.362+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:10.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:10.362+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:10.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:10.205+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:10.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:10.205+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:10.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:10.205+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:10.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:10.205+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:10.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:10.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:10.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:10.471+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:11.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:11.466+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:11.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:11.466+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:11.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:11.466+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:11.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:11.466+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:11.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:11.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:11.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:11.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:11.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:11.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:11.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:11.247+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:11.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:11.247+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:11.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:11.247+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:11.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:11.247+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:12.415 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:12.004+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8101 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:12.415 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:12.249+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:12.415 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:12.249+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:12.415 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:12.249+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:12.415 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:12.250+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:12.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:12.428+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:12.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:12.428+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:12.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:12.428+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:12.501+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:12.501+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:12.501+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:12.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:12.502+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:13.687 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:13.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:13.687 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:13.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:13.687 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:13.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:13.688 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:13.482+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:13.688 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:13.482+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:13.688 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:13.482+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:13.688 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:13.482+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:13.688 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:13.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:13.688 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:13.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:13.689 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:13.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:13.689 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:13.267+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:14.508 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:14.401+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:14.508 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:14.401+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:14.508 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:14.401+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:14.509 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:14.230+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:14.509 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:14.230+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:14.509 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:14.230+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:14.509 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:14.230+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:14.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:14.522+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:14.522+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:14.522+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:14.522+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:15.474 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:15.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:15.474 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:15.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:15.474 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:15.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:15.474 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:15.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:15.475 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:15.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:15.475 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:15.237+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:15.475 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:15.237+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:15.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:15.488+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:15.488+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:15.488+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:15.488+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:16.461+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:16.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:16.461+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:16.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:16.461+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:16.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:16.462+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:16.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:16.347+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:16.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:16.347+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:16.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:16.347+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:16.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:16.229+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:16.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:16.229+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:16.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:16.229+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:16.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:16.229+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:17.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:17.006+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8158 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:17.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:17.440+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:17.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:17.440+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:17.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:17.440+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:17.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:17.440+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:17.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:17.302+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:17.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:17.302+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:17.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:17.302+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:17.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:17.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:17.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:17.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:17.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:17.267+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:17.709 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:17.267+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:18.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:18.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:18.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:18.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:18.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:18.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:18.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:18.487+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:18.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:18.271+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:18.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:18.271+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:18.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:18.271+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:18.295+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:18.295+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:18.295+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:18.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:18.295+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:19.500 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:19.254+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:19.500 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:19.254+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:19.500 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:19.254+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:19.501 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:19.247+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:19.501 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:19.247+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:19.501 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:19.247+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:19.501 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:19.247+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:19.514+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:19.514+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:19.514+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:19.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:19.514+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:20.454 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:20.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:20.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:20.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:20.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:20.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:20.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:20.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:20.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:20.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:20.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:20.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:20.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:20.200+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:20.468+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:20.468+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:20.468+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:20.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:20.468+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:21.282 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:21.232+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:21.282 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:21.232+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:21.282 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:21.232+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:21.282 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:21.232+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:21.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:21.294+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:21.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:21.294+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:21.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:21.294+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:21.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:21.488+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:21.488+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:21.488+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:22.270 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:22.007+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8217 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:22.271 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:22.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:22.271 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:22.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:22.271 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:22.233+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:22.271 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:22.233+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:22.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:22.284+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:22.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:22.284+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:22.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:22.284+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:22.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:22.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:22.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:22.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:22.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:22.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:22.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:22.518+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:23.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:23.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:23.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:23.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:23.492+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:23.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:23.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:23.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:23.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:23.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:23.273+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:23.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:23.273+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:24.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:24.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:24.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:24.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:24.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:24.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:24.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:24.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:24.291+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:24.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:24.291+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:24.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:24.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:24.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:24.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:24.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:24.471+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:24.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:24.471+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:25.493 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:25.228+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:25.494 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:25.228+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:25.494 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:25.228+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:25.494 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:25.323+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:25.494 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:25.323+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:25.494 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:25.323+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:25.494 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:25.323+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:25.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:25.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:25.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:25.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:25.507+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:26.447 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:26.179+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:26.447 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:26.179+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:26.447 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:26.179+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:26.448 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:26.354+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:26.448 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:26.354+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:26.448 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:26.354+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:26.448 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:26.354+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:26.461+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:26.461+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:26.461+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:26.461+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:27.324 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:27.008+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8274 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:27.324 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:27.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:27.324 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:27.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:27.324 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:27.203+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:27.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:27.493+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:27.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:27.493+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:27.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:27.493+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:27.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:27.493+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:27.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:27.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:27.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:27.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:27.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:27.338+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:27.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:27.338+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:28.449 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:28.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:28.450 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:28.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:28.450 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:28.186+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:28.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:28.363+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:28.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:28.363+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:28.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:28.363+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:28.451 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:28.363+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:28.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:28.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:28.464+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:28.464+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:29.392 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:29.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:29.393 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:29.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:29.393 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:29.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:29.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:29.493+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:29.493+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:29.493+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:29.493+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:29.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:29.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:29.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:29.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:29.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:29.406+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:29.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:29.406+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:30.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:30.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:30.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:30.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:30.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:30.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:30.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:30.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:30.379+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:30.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:30.379+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:30.470+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:30.470+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:30.470+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:30.470+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:31.427 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:31.158+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:31.427 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:31.158+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:31.428 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:31.158+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:31.428 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:31.398+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:31.428 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:31.398+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:31.428 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:31.398+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:31.428 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:31.398+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:31.441+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:31.441+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:31.441+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:31.441+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:32.433 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:32.010+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8333 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:32.433 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:32.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:32.434 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:32.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:32.434 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:32.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:32.457+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:32.457+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:32.457+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:32.457+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:32.447+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:32.447+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:32.447+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:32.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:32.447+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:33.418 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:33.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:33.419 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:33.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:33.419 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:33.177+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:33.432+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:33.432+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:33.432+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:33.432+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:33.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:33.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:33.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:33.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:33.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:33.435+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:33.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:33.435+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:34.450 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:34.187+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:34.451 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:34.187+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:34.451 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:34.187+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:34.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:34.424+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:34.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:34.424+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:34.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:34.424+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:34.451 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:34.424+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:34.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:34.465+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:34.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:34.465+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:34.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:34.465+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:34.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:34.465+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:35.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:35.190+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:35.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:35.190+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:35.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:35.190+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:35.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:35.423+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:35.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:35.423+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:35.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:35.423+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:35.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:35.423+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:35.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:35.435+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:35.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:35.435+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:35.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:35.435+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:35.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:35.435+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:36.432 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:36.191+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:36.433 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:36.191+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:36.433 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:36.191+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:36.446+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:36.446+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:36.446+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:36.446+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:36.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:36.450+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:36.450+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:36.450+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:36.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:36.450+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:37.415 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:37.010+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8390 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:37.416 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:37.180+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:37.416 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:37.180+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:37.416 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:37.180+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:37.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:37.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:37.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:37.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:37.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:37.467+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:37.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:37.467+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:37.428+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:37.428+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:37.428+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:37.428+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:38.402 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-04-15T23:15:38.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T23:15:38.408 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-15T23:15:38.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T23:15:38.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T23:15:38.413 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-04-15T23:15:38.435 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-15T23:15:38.446 INFO:tasks.cephadm:Teardown begin 2024-04-15T23:15:38.446 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-04-15T23:15:38.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:15:38.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:15:38.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:38.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:38.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:38.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:38.463 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:38.155+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:38.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:38.450+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:38.450+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:38.450+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:38.450+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:38.510+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:38.510+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:38.510+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:38.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:38.510+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:39.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:39.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:39.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:39.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:39.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:39.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:39.410+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:39.410+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:39.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:39.410+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:39.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:39.410+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:39.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:39.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:39.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:39.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:39.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:39.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:39.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:39.532+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:40.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:40.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:40.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:40.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:40.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:40.150+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:40.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:40.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:40.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:40.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:40.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:40.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:40.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:40.398+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:40.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:40.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:40.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:40.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:40.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:40.526+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:40.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:40.526+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:41.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:41.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:41.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:41.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:41.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:41.358+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:41.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:41.176+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:41.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:41.176+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:41.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:41.176+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:41.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:15:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:41.477+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:41.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:41.477+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:41.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:41.477+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:41.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:41.477+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:42.386 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:42.011+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8449 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:42.386 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:42.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:42.386 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:42.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:42.386 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:42.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:42.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:42.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:42.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:42.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:42.399+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:42.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:42.497+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:42.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:42.497+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:42.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:42.497+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:42.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:42.497+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:43.519 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:43.243+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:43.519 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:43.243+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:43.519 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:43.243+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:43.519 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:43.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:43.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:43.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:43.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:43.444+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:43.520 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:43.444+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:43.895 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:43.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:43.895 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:43.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:43.895 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:43.532+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:43.895 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:43.532+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:44.540 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:44.239+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:44.540 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:44.239+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:44.540 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:44.239+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:44.540 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:44.489+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:44.540 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:44.489+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:44.541 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:44.489+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:44.541 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:44.489+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:44.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:44.553+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:44.553+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:44.553+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:44.553+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:45.559 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:45.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:45.559 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:45.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:45.559 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:45.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:45.559 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:45.496+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:45.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:45.507+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:45.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:45.507+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:45.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:45.507+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:45.560 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:45.507+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:45.560 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:45.267+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:45.561 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:45.267+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:45.561 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:45.267+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:46.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:46.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:46.642 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:46.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:46.642 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:46.487+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:46.642 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:46.487+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:46.642 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:46.252+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:46.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:46.252+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:46.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:46.252+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:46.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:46.476+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:46.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:46.476+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:46.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:46.476+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:46.643 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:46.477+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:47.458 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:47.013+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8506 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:47.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:47.237+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:47.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:47.237+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:47.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:47.237+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:47.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:47.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:47.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:47.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:47.491+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:47.510+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:47.510+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:47.510+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:47.510+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:48.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:48.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:48.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:48.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:48.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:48.490+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:48.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:48.243+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:48.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:48.243+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:48.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:48.243+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:48.475+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:48.475+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:48.475+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:48.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:48.475+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:49.500 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:49.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:49.500 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:49.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:49.500 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:49.233+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:49.500 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:49.462+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:49.500 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:49.462+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:49.501 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:49.462+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:49.501 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:49.462+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:49.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:49.513+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:49.513+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:49.513+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:49.513+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:50.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:15:50.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:15:50.528 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:50.274+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:50.528 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:50.274+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:50.528 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:50.274+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:50.528 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:50.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:50.528 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:50.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:50.529 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:50.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:50.529 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:50.509+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:50.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:50.541+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:50.541+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:50.541+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:50.541+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:51.483 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:51.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:51.483 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:51.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:51.484 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:51.232+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:51.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:51.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:51.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:51.496+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:51.576+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:51.576+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:51.576+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:51.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:51.576+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:52.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:52.014+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8565 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:52.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:52.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:52.244+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:52.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:52.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:52.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:52.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:52.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:52.536+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:52.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:52.536+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:52.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:52.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:52.530+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:52.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:52.530+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:53.505 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:53.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:53.506 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:53.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:53.506 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:53.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:53.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:15:53.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:53.519+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:53.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:53.519+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:53.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:53.519+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:53.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:53.519+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:53.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:53.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:53.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:53.572+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:54.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:54.207+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:54.207+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:54.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:54.207+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:54.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:54.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:54.531+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:54.531+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:54.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:54.544+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:54.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:54.544+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:54.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:54.544+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:54.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:54.544+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:55.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:55.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:55.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:55.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:55.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:55.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:55.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:55.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:55.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:55.571+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:55.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:55.571+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:55.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:55.571+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:55.559+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:55.559+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:55.559+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:55.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:55.559+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:56.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:56.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:56.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:56.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:56.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:56.202+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:56.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:56.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:56.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:56.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:56.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:56.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:56.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:56.586+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:56.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:56.574+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:56.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:56.574+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:56.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:56.574+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:56.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:56.574+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:57.457 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:15:57.016+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8622 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:15:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:57.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:57.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:57.458 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:57.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:57.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:57.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:57.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:57.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:57.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:57.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:57.570+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:57.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:57.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:57.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:57.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:57.572+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:58.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:58.192+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:58.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:58.192+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:58.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:58.192+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:58.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:58.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:58.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:58.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:58.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:58.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:58.585+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:58.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:58.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:58.544+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:58.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:58.544+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:59.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:59.172+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:59.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:59.172+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:59.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:15:59.172+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:15:59.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:59.625+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:59.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:59.625+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:59.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:59.625+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:15:59.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:15:59.625+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:15:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:59.576+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:59.576+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:59.576+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:15:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:15:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:15:59.576+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:00.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:00.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:00.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:00.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:00.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:00.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:00.920 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:00.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:00.920 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:00.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:00.920 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:00.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:00.920 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:00.614+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:00.921 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:00.635+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:00.921 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:00.635+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:00.921 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:00.635+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:00.921 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:00.635+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:01.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:01.194+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:01.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:01.194+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:01.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:01.194+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:01.658 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:01.594+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:01.594+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:01.594+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:01.659 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:01.594+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:01.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:01.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:01.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:01.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:01.659 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:01.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:01.660 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:01.671+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:02.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:02.018+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8681 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:02.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:02.204+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:02.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:02.204+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:02.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:02.204+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:02.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:02.708+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:02.708+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:02.708+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:02.708+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:02.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:02.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:02.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:02.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:02.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:02.588+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:03.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:03.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:03.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:03.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:03.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:03.174+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:03.708+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:03.708+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:03.708+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:03.708+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:03.637+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:03.637+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:03.637+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:03.637+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:04.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:04.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:04.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:04.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:04.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:04.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:04.678+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:04.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:04.678+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:04.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:04.678+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:04.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:04.678+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:04.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:04.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:04.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:04.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:04.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:04.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:04.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:04.592+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:05.541 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:05.235+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:05.542 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:05.235+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:05.542 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:05.235+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:05.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:05.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:05.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:05.643+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:05.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:05.643+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:05.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:05.643+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:05.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:05.643+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:05.555+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:05.555+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:05.555+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:05.556+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:06.565 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:06.270+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:06.565 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:06.270+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:06.565 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:06.270+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:06.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:06.630+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:06.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:06.630+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:06.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:06.630+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:06.960 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:06.630+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:06.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:06.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:06.579+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:06.960 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:06.579+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:07.268 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:07.019+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8738 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:07.585 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:07.282+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:07.586 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:07.282+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:07.586 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:07.282+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:07.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:07.654+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:07.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:07.654+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:07.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:07.654+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:07.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:07.654+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:07.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:07.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:07.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:07.599+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:08.587 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:08.314+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:08.587 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:08.314+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:08.587 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:08.314+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:08.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:08.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:08.669+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:08.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:08.669+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:08.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:08.669+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:08.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:08.669+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:08.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:08.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:08.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:08.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:08.599+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:08.599+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:09.574 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:09.311+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:09.574 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:09.311+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:09.574 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:09.311+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:09.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:09.624+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:09.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:09.624+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:09.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:09.624+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:09.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:09.624+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:09.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:09.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:09.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:09.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:09.588+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:10.592 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:10.307+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:10.592 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:10.307+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:10.593 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:10.307+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:10.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:10.606+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:10.606+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:10.606+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:10.606+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:10.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:10.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:10.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:10.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:10.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:11.560 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:11.259+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:11.560 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:11.259+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:11.560 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:11.259+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:11.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:11.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:11.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:11.711 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:16:11.711 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-15T23:16:11.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:11.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:11.849 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:11.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:11.850 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:11.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:11.850 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:11.650+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:11.850 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:11.650+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:11.850 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:11.574+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:11.851 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:11.574+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:11.851 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:11.574+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:11.851 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:11.574+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:12.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:12.021+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8798 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:12.577 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:12.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:12.577 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:12.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:12.577 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:12.289+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:12.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:12.653+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:12.653+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:12.653+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:12.653+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:12.591+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:12.591+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:12.591+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:12.591+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:13.558 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:13.274+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:13.558 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:13.274+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:13.558 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:13.274+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:13.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:13.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:13.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:13.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:13.684+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:13.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:13.571+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:13.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:13.571+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:13.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:13.571+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:13.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:13.572+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:14.548 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:14.287+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:14.548 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:14.287+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:14.548 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:14.287+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:14.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:14.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:14.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:14.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:14.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:14.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:14.562+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:14.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:14.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:14.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:14.699+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:15.531 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:15.261+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:15.531 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:15.261+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:15.531 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:15.261+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:15.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:15.738+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:15.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:15.738+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:15.738+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:15.738+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:15.545+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:15.545+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:15.545+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:15.545+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:16.518 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:16.240+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:16.519 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:16.240+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:16.519 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:16.240+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:16.891 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:16.763+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:16.891 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:16.763+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:16.892 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:16.763+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:16.892 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:16.763+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:16.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:16.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:16.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:16.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:16.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:16.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:16.892 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:16.532+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:17.202 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:17.022+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8856 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:17.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:17.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:17.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:17.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:17.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:17.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:17.736 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:17.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:17.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:17.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:17.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:17.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:17.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:17.492+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:18.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:17.750+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:18.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:17.750+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:18.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:17.750+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:18.174 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:17.750+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:18.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:18.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:18.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:18.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:18.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:18.188+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:18.745 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:18.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:18.745 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:18.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:18.745 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:18.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:18.745 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:18.505+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:19.130 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:18.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:19.130 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:18.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:19.130 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:18.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:19.130 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:18.759+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:19.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:19.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:19.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:19.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:19.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:19.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:19.755 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:19.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:19.756 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:19.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:19.756 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:19.490+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:19.756 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:19.490+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:20.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:19.769+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:20.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:19.769+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:20.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:19.769+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:20.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:19.770+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:20.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:20.133+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:20.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:20.133+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:20.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:20.133+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:20.752 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:20.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:20.752 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:20.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:20.753 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:20.509+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:20.753 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:20.509+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:21.143 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:20.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:21.143 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:20.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:21.143 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:20.766+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:21.143 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:20.766+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:21.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:21.157+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:21.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:21.157+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:21.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:21.157+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:21.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:21.722+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:21.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:21.722+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:21.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:21.722+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:21.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:21.722+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:21.933 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:21.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:21.934 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:21.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:21.934 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:21.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:21.934 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:21.532+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:22.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:22.023+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8916 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:22.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:22.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:22.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:22.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:22.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:22.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:22.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:22.733+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:22.733+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:22.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:22.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:22.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:22.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:22.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:22.491+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:22.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:22.491+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:23.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:23.115+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:23.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:23.115+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:23.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:23.115+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:23.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:23.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:23.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:23.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:23.747+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:23.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:23.747+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:23.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:23.539+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:23.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:23.539+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:23.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:23.539+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:23.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:23.539+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:24.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:24.129+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:24.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:24.129+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:24.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:24.129+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:24.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:24.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:24.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:24.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:24.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:24.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:24.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:24.724+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:24.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:24.576+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:24.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:24.576+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:24.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:24.576+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:24.959 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:24.576+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:25.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:25.091+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:25.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:25.091+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:25.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:25.091+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:25.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:25.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:25.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:25.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:25.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:25.725+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:25.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:25.726+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:25.587+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:25.587+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:25.587+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:25.587+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:26.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:26.121+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:26.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:26.121+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:26.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:26.121+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:26.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:26.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:26.578+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:26.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:26.578+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:26.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:26.578+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:26.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:26.578+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:26.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:26.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:26.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:26.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:26.721+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:27.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:27.024+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 8970 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:27.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:27.073+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:27.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:27.073+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:27.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:27.073+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:27.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:27.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:27.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:27.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:27.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:27.562+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:27.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:27.562+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:27.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:27.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:27.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:27.707+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:28.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:28.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:28.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:28.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:28.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:28.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:28.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:28.736+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:28.736+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:28.736+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:28.736+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:28.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:28.611+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:28.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:28.611+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:28.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:28.611+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:28.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:28.611+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:29.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:29.039+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:29.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:29.039+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:29.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:29.039+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:29.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:29.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:29.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:29.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:29.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:29.693+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:29.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:29.628+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:29.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:29.628+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:29.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:29.628+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:29.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:29.628+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:30.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:30.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:30.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:30.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:30.667+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:30.667+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:30.667+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:30.667+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:30.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:30.651+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:30.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:30.651+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:30.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:30.651+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:30.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:30.651+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:31.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:30.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:31.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:30.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:31.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:30.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:31.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:31.640+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:31.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:31.640+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:31.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:31.640+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:31.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:31.640+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:31.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:31.633+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:31.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:31.633+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:31.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:31.633+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:31.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:31.633+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:32.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:32.025+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9022 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:32.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:31.981+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:32.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:31.981+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:32.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:31.981+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:32.938 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:32.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:32.938 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:32.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:32.938 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:32.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:32.938 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:32.632+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:32.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:32.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:32.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:32.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:32.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:32.665+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:32.939 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:32.665+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:33.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:32.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:33.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:32.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:33.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:32.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:33.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:33.627+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:33.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:33.627+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:33.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:33.627+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:33.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:33.627+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:33.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:33.615+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:33.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:33.615+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:33.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:33.615+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:33.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:33.615+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:34.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:33.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:34.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:33.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:34.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:33.982+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:34.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:34.617+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:34.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:34.617+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:34.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:34.617+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:34.903 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:34.617+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:34.903 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:34.643+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:34.904 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:34.643+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:34.904 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:34.643+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:34.904 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:34.643+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:34.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:34.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:35.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:34.964+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:35.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:35.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:35.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:35.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:35.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:35.595+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:35.692+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:35.692+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:35.692+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:35.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:35.692+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:36.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:35.991+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:36.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:35.991+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:36.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:35.991+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:36.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:36.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:36.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:36.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:36.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:36.585+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:36.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:36.585+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:36.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:36.711+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:36.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:36.711+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:36.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:36.711+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:36.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:36.711+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:37.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:37.026+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9074 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:37.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:37.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:37.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:37.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:37.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:37.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:37.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:37.586+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:37.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:37.586+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:37.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:37.586+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:37.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:37.586+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:37.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:37.684+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:37.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:37.684+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:37.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:37.684+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:37.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:37.684+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:38.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:37.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:38.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:37.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:38.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:37.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:38.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:38.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:38.950 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:38.668+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:38.950 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:38.668+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:38.950 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:38.668+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:38.950 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:38.668+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:38.951 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:38.629+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:38.951 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:38.629+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:38.951 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:38.629+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:38.951 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:38.629+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:38.962+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:38.962+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:39.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:38.962+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:39.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:39.674+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:39.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:39.674+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:39.947 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:39.674+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:39.948 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:39.674+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:39.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:39.657+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:39.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:39.657+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:39.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:39.657+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:39.948 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:39.657+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:39.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:39.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:39.961+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:40.941 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:40.668+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:40.941 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:40.668+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:40.941 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:40.668+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:40.941 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:40.668+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:40.941 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:40.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:40.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:40.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:40.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:40.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:40.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:40.681+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:40.954+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:40.954+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:41.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:40.954+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:41.921 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:41.729+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:41.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:41.729+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:41.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:41.729+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:41.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:41.729+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:41.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:41.928+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:41.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:41.928+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:41.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:41.928+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:41.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:41.701+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:41.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:41.701+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:41.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:41.701+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:41.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:41.701+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:42.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:42.028+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9128 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:42.941 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:42.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:42.941 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:42.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:42.941 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:42.682+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:42.941 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:42.682+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:42.942 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:42.746+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:42.942 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:42.746+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:42.942 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:42.746+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:42.942 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:42.746+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:43.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:42.955+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:43.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:42.955+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:43.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:42.955+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:43.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:43.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:43.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:43.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:43.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:43.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:43.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:43.715+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:43.781+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:43.781+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:43.781+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:43.781+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:44.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:43.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:44.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:43.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:44.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:43.988+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:44.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:44.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:44.934 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:44.737+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:44.934 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:44.737+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:44.935 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:44.737+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:44.935 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:44.737+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:44.935 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:44.695+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:44.935 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:44.695+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:44.935 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:44.695+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:44.935 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:44.695+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:44.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:44.990 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:16:44.991 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-15T23:16:44.993 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/60ff499a-fb7a-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608/remote/smithi037/crash 2024-04-15T23:16:44.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:44.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:45.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:44.948+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:45.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:44.948+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:45.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:44.948+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:45.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:45.731+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:45.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:45.731+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:45.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:45.731+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:45.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:45.731+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:45.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:45.689+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:45.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:45.689+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:45.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:45.689+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:45.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:45.689+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:46.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:45.985+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:46.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:45.985+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:46.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:45.985+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:46.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:46.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:46.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:46.740+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:46.740+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:46.670+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:46.670+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:46.670+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:46.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:46.670+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:47.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:47.029+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9180 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:47.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:46.987+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:47.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:46.987+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:47.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:46.987+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:47.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:47.709+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:47.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:47.709+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:47.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:47.709+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:47.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:47.709+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:47.696+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:47.696+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:47.696+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:47.697+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:48.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:16:48.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:47.998+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:48.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:47.998+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:48.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:47.998+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:48.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:48.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:48.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:48.662+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:48.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:48.662+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:48.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:48.662+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:48.719+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:48.719+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:48.719+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:48.719+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:49.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:49.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:49.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:49.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:49.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:49.042+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:49.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:49.619+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:49.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:49.619+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:49.619+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:49.619+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:49.702+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:49.702+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:49.702+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:49.702+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:50.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:50.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:50.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:50.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:50.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:50.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:50.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:50.660+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:50.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:50.660+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:50.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:50.660+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:50.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:50.660+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:50.751+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:50.751+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:50.751+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:50.751+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:51.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:51.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:51.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:51.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:51.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:51.031+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:51.687+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:51.687+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:51.687+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:51.687+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:51.797+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:51.797+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:51.797+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:51.797+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:52.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:52.032+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9232 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:52.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:52.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:52.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:52.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:52.998 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:52.784+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:52.999 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:52.784+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:52.999 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:52.784+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:52.999 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:52.784+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:52.999 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:52.732+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:52.999 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:52.732+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:52.999 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:52.732+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:53.000 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:52.732+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:53.335 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:53.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:53.335 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:53.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:53.335 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:53.012+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:54.046 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:53.752+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:54.046 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:53.752+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:54.046 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:53.752+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:54.046 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:53.752+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:54.047 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:53.777+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:54.047 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:53.777+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:54.047 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:53.777+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:54.047 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:53.777+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:54.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:54.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:54.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:54.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:54.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:54.060+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:55.044 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:54.733+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:55.044 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:54.733+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:55.044 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:54.733+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:55.044 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:54.734+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:55.044 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:54.791+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:55.045 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:54.792+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:55.045 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:54.792+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:55.045 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:54.792+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:55.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:55.058+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:55.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:55.058+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:55.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:55.058+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:56.039 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:55.761+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:56.040 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:55.761+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:56.040 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:55.761+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:56.040 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:55.761+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:56.040 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:55.770+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:56.040 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:55.770+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:56.040 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:55.770+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:56.041 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:55.770+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:56.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:56.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:56.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:56.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:56.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:56.053+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:57.019 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:56.781+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:57.019 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:56.781+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:57.019 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:56.781+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:57.020 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:56.781+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:57.020 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:56.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:57.020 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:56.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:57.020 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:56.724+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:57.020 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:56.725+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:57.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:16:57.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:16:57.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:16:57.033+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9284 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:16:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:57.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:57.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:57.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:57.036+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:57.795+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:57.795+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:57.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:57.795+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:57.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:57.795+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:57.687+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:57.687+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:57.687+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:57.687+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:58.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:58.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:58.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:58.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:58.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:58.019+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:58.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:58.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:58.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:58.790+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:58.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:58.791+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:58.671+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:58.671+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:58.671+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:58.671+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:59.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:59.006+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:59.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:59.006+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:59.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:59.006+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:16:59.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:59.752+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:59.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:59.752+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:59.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:59.752+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:16:59.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:16:59.752+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:16:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:59.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:59.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:59.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:16:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:16:59.715+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:00.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:00.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:59.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:00.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:59.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:00.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:16:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:16:59.989+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:00.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:00.713+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:00.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:00.713+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:00.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:00.713+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:00.713+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:00.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:00.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:00.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:00.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:00.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:00.710+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:00.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:00.710+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:01.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:00.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:01.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:00.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:01.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:00.976+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:01.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:01.945+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:01.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:01.945+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:01.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:01.945+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:01.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:01.718+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:01.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:01.718+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:01.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:01.718+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:01.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:01.718+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:01.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:01.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:01.741+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:01.741+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:02.264 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:02.034+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9337 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:02.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:02.688+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:02.688+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:02.688+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:02.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:02.688+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:02.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:02.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:02.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:02.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:02.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:02.699+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:02.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:02.699+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:03.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:02.974+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:03.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:02.974+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:03.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:02.974+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:03.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:03.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:03.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:03.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:03.642+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:03.742+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:03.742+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:03.742+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:03.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:03.742+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:04.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:03.978+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:04.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:03.978+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:04.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:03.978+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:04.614+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:04.614+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:04.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:04.614+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:04.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:04.614+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:04.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:04.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:04.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:04.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:04.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:04.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:04.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:04.759+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:05.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:05.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:05.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:05.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:05.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:05.020+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:05.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:05.648+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:05.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:05.648+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:05.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:05.648+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:05.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:05.648+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:05.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:05.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:05.783+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:05.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:05.783+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:06.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:06.052+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:06.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:06.052+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:06.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:06.052+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:06.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:06.677+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:06.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:06.677+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:06.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:06.677+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:06.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:06.677+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:06.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:06.761+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:06.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:06.761+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:06.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:06.761+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:06.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:06.761+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:07.305 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:07.036+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9389 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:07.306 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:07.063+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:07.306 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:07.064+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:07.306 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:07.064+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:07.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:07.720+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:07.720+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:07.720+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:07.720+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:07.801+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:07.801+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:07.801+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:07.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:07.801+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:08.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:08.087+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:08.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:08.087+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:08.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:08.087+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:08.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:08.698+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:08.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:08.698+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:08.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:08.698+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:08.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:08.698+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:08.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:08.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:08.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:08.759+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:08.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:08.759+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:09.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:09.084+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:09.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:09.084+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:09.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:09.084+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:09.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:09.681+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:09.681+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:09.681+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:09.681+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:09.794+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:09.794+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:09.794+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:09.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:09.794+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:10.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:10.107+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:10.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:10.107+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:10.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:10.107+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:10.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:10.679+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:10.679+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:10.679+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:10.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:10.679+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:10.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:10.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:10.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:10.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:10.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:10.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:10.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:10.758+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:11.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:11.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:11.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:11.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:11.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:11.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:11.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:11.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:11.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:11.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:11.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:11.661+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:11.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:11.661+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:11.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:11.713+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:11.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:11.713+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:11.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:11.713+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:11.959 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:11.713+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:12.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:12.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:12.347 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:12.037+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9442 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:12.347 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:12.138+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:12.347 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:12.138+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:12.347 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:12.138+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:12.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:12.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:12.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:12.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:12.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:12.632+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:12.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:12.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:12.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:12.748+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:12.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:12.748+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:13.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:13.106+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:13.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:13.106+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:13.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:13.106+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:13.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:13.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:13.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:13.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:13.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:13.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:13.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:13.671+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:13.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:13.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:13.711+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:13.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:13.711+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:14.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:14.136+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:14.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:14.136+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:14.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:14.136+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:14.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:14.665+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:14.665+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:14.665+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:14.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:14.665+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:14.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:14.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:14.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:14.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:14.703+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:14.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:14.703+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:15.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:15.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:15.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:15.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:15.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:15.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:15.181+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:15.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:15.672+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:15.672+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:15.672+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:15.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:15.672+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:15.687+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:15.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:15.687+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:15.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:15.687+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:15.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:15.687+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:16.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:16.160+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:16.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:16.160+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:16.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:16.160+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:16.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:16.639+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:16.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:16.639+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:16.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:16.639+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:16.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:16.639+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:16.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:16.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:16.721+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:16.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:16.721+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:17.388 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:17.038+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9494 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:17.388 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:17.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:17.388 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:17.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:17.388 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:17.151+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:17.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:17.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:17.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:17.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:17.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:17.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:17.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:17.613+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:17.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:17.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:17.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:17.684+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:18.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:18.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:18.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:18.271 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:17:18.271 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-15T23:17:18.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:18.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:18.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:18.145+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:18.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:18.145+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:18.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:18.145+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:18.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:18.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:18.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:18.568+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:18.568+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:18.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:18.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:18.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:18.681+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:19.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:19.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:19.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:19.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:19.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:19.108+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:19.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:19.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:19.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:19.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:19.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:19.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:19.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:19.584+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:19.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:19.649+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:19.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:19.649+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:19.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:19.649+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:19.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:19.649+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:20.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:20.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:20.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:20.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:20.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:20.144+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:20.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:20.575+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:20.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:20.575+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:20.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:20.575+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:20.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:20.575+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:20.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:20.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:20.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:20.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:20.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:20.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:20.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:20.634+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:21.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:21.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:21.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:21.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:21.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:21.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:21.171+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:21.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:21.615+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:21.615+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:21.615+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:21.615+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:21.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:21.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:21.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:21.648+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:22.429 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:22.039+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9547 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:22.429 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:22.200+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:22.429 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:22.200+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:22.429 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:22.200+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:22.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:22.650+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:22.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:22.650+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:22.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:22.650+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:22.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:22.650+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:22.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:22.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:22.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:22.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:22.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:22.670+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:22.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:22.671+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:23.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:23.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:23.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:23.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:23.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:23.210+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:23.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:23.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:23.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:23.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:23.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:23.671+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:23.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:23.671+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:23.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:23.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:23.634+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:23.634+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:24.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:24.211+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:24.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:24.211+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:24.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:24.211+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:24.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:24.636+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:24.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:24.636+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:24.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:24.636+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:24.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:24.636+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:24.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:24.640+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:24.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:24.640+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:24.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:24.640+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:24.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:24.640+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:25.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:25.195+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:25.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:25.195+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:25.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:25.195+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:25.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:25.666+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:25.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:25.666+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:25.666+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:25.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:25.666+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:25.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:25.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:25.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:25.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:25.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:25.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:25.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:25.595+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:26.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:26.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:26.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:26.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:26.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:26.215+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:26.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:26.677+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:26.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:26.677+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:26.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:26.677+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:26.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:26.677+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:26.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:26.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:26.590+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:26.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:26.590+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:27.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:27.040+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9598 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:27.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:27.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:27.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:27.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:27.457 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:27.225+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:27.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:27.699+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:27.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:27.699+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:27.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:27.699+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:27.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:27.699+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:27.564+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:27.564+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:27.564+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:27.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:27.564+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:28.531 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:28.241+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:28.588 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:28.241+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:28.588 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:28.241+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:28.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:28.666+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:28.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:28.666+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:28.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:28.666+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:28.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:28.666+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:28.545+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:28.545+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:28.545+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:28.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:28.545+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:29.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:29.208+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:29.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:29.208+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:29.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:29.208+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:29.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:29.631+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:29.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:29.631+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:29.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:29.631+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:29.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:29.631+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:29.504+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:29.504+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:29.504+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:29.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:29.504+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:30.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:30.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:30.252+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:30.252+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:30.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:30.252+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:30.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:30.629+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:30.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:30.629+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:30.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:30.629+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:30.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:30.629+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:30.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:30.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:30.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:30.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:30.549+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:31.548 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:31.286+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:31.548 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:31.286+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:31.548 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:31.286+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:31.926 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:31.607+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:31.927 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:31.607+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:31.927 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:31.607+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:31.927 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:31.607+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:31.932 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:31.561+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:31.932 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:31.561+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:31.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:31.561+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:31.933 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:31.561+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:32.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:32.042+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9650 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:32.582 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:32.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:32.583 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:32.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:32.583 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:32.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:32.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:32.524+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:32.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:32.524+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:32.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:32.524+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:32.583 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:32.524+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:32.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:32.596+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:32.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:32.596+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:32.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:32.596+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:32.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:32.596+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:33.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:33.533 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:33.353+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:33.533 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:33.353+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:33.533 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:33.353+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:33.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:33.546+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:33.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:33.546+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:33.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:33.546+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:33.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:33.546+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:33.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:33.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:33.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:33.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:33.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:33.570+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:33.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:33.570+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:34.607 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:34.368+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:34.607 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:34.368+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:34.607 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:34.368+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:34.608 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:34.548+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:34.608 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:34.548+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:34.608 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:34.548+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:34.608 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:34.548+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:34.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:34.621+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:34.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:34.621+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:34.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:34.621+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:34.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:34.621+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:35.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:35.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:35.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:35.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:35.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:35.595+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:35.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:35.595+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:35.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:35.367+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:35.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:35.367+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:35.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:35.367+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:35.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:35.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:35.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:35.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:35.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:35.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:35.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:35.518+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:36.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:36.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:36.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:36.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:36.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:36.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:36.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:36.592+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:36.402+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:36.402+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:36.402+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:36.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:36.503+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:36.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:36.503+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:36.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:36.503+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:36.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:36.503+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:37.362 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:37.043+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9701 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:37.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:37.603+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:37.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:37.603+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:37.706 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:37.603+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:37.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:37.606+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:37.376+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:37.376+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:37.376+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:37.553+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:37.553+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:37.553+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:37.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:37.553+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:38.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:38.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:38.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:38.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:38.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:38.352+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:38.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:38.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:38.572+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:38.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:38.572+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:38.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:38.513+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:38.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:38.513+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:38.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:38.513+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:38.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:38.513+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:39.600 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:39.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:39.601 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:39.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:39.601 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:39.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:39.601 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:39.493+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:39.601 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:39.310+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:39.601 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:39.310+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:39.601 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:39.310+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:39.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:39.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:39.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:39.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:39.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:39.614+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:39.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:39.615+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:40.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:40.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:40.642 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:40.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:40.642 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:40.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:40.642 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:40.541+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:40.642 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:40.329+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:40.642 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:40.329+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:40.643 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:40.329+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:40.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:40.655+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:40.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:40.655+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:40.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:40.655+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:40.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:40.655+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:41.640 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:41.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:41.640 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:41.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:41.640 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:41.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:41.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:41.508+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:41.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:41.508+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:41.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:41.508+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:41.641 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:41.508+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:41.654+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:41.654+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:41.654+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:41.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:41.654+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:42.349 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:42.046+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9754 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:42.622 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:42.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:42.623 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:42.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:42.623 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:42.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:42.623 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:42.552+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:42.623 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:42.552+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:42.624 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:42.552+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:42.624 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:42.552+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:42.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:42.636+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:42.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:42.636+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:42.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:42.636+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:42.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:42.636+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:43.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:43.554+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:43.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:43.554+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:43.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:43.554+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:43.619 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:43.554+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:43.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:43.328+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:43.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:43.328+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:43.620 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:43.328+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:43.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:43.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:43.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:43.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:43.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:43.633+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:43.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:43.633+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:44.550 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:44.287+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:44.550 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:44.287+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:44.550 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:44.287+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:44.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:44.564+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:44.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:44.564+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:44.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:44.564+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:44.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:44.564+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:44.584+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:44.584+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:44.584+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:44.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:44.585+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:45.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:45.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:45.580 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:45.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:45.580 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:45.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:45.580 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:45.317+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:45.942 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:45.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:45.943 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:45.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:45.943 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:45.604+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:45.943 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:45.604+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:45.943 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:45.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:45.943 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:45.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:45.943 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:45.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:45.944 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:45.594+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:46.618 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:46.304+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:46.618 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:46.304+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:46.618 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:46.304+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:46.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:46.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:46.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:46.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:46.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:46.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:46.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:46.632+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:46.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:46.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:46.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:46.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:46.639+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:47.286 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:47.047+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9805 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:47.605 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:47.300+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:47.605 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:47.300+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:47.606 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:47.300+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:47.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:47.653+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:47.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:47.653+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:47.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:47.653+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:47.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:47.653+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:47.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:47.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:47.619+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:47.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:47.619+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:48.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:48.614 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:48.349+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:48.614 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:48.349+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:48.614 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:48.349+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:48.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:48.663+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:48.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:48.663+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:48.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:48.663+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:48.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:48.663+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:48.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:48.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:48.628+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:48.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:48.628+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:49.593 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:49.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:49.594 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:49.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:49.595 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:49.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:49.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:49.657+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:49.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:49.657+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:49.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:49.657+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:49.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:49.657+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:49.608+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:49.608+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:49.608+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:49.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:49.608+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:50.578 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:50.342+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:50.579 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:50.342+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:50.579 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:50.342+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:50.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:50.645+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:50.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:50.645+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:50.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:50.645+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:50.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:50.645+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:50.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:50.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:50.592+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:50.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:50.592+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:51.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:51.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:51.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:51.550 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:17:51.551 INFO:tasks.cephadm:Removing cluster... 2024-04-15T23:17:51.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:17:51.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:17:51.604 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:51.358+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:51.604 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:51.358+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:51.604 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:51.358+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:51.647+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:51.647+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:51.647+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:51.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:51.647+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:51.618+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:51.618+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:51.618+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:51.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:51.618+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:52.363 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:52.048+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9857 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:52.627 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:52.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:52.628 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:52.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:52.628 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:52.377+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:52.628 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:52.625+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:52.628 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:52.625+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:52.628 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:52.625+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:52.629 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:52.625+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:52.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:52.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:52.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:52.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:52.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:52.642+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:52.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:52.642+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:53.666 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:53.351+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:53.667 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:53.351+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:53.667 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:53.351+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:53.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:53.601+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:53.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:53.601+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:53.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:53.601+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:53.668 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:53.601+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:53.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:53.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:53.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:53.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:53.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:53.681+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:53.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:53.681+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:54.598 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:54.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:54.598 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:54.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:54.598 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:54.339+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:54.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:17:54.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:54.612+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:54.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:54.612+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:54.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:54.612+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:54.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:54.612+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:54.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:54.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:54.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:54.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:54.648+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:54.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:54.648+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:55.649 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:55.340+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:55.650 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:55.340+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:55.650 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:55.340+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:55.650 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:55.572+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:55.650 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:55.572+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:55.650 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:55.572+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:55.650 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:55.572+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:55.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:55.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:55.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:55.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:55.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:55.664+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:55.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:55.664+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:56.605 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:56.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:56.606 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:56.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:56.606 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:56.346+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:56.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:56.543+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:56.606 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:56.543+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:56.607 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:56.543+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:56.607 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:56.543+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:56.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:56.620+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:56.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:56.620+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:56.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:56.620+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:56.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:56.620+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:57.314 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:17:57.050+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9908 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:17:57.647 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:57.329+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:57.647 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:57.329+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:57.647 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:57.329+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:57.647 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:57.576+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:57.648 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:57.576+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:57.648 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:57.576+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:57.648 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:57.576+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:57.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:57.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:57.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:57.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:57.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:57.661+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:57.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:57.661+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:58.673 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:58.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:58.673 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:58.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:58.674 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:58.298+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:58.674 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:58.537+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:58.674 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:58.537+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:58.677 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:58.537+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:58.677 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:58.537+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:58.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:58.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:58.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:58.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:58.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:58.688+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:58.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:58.688+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:59.666 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:59.318+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:59.666 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:59.318+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:59.666 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:17:59.318+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:17:59.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:59.543+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:59.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:59.543+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:59.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:59.543+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:17:59.667 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:17:59.544+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:17:59.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:59.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:59.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:59.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:59.680+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:17:59.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:17:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:17:59.680+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:00.579 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:00.337+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:00.580 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:00.337+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:00.580 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:00.337+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:00.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:00.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:00.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:00.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:00.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:00.693+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:00.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:00.693+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:00.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:00.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:00.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:00.594+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:01.639 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:01.370+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:01.639 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:01.370+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:01.640 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:01.370+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:01.640 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:01.559+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:01.640 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:01.559+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:01.640 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:01.559+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:01.640 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:01.559+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:01.654+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:01.654+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:01.654+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:01.958 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:01.654+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:02.391 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:02.051+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 9961 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:02.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:02.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:02.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:02.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:02.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:02.406+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:02.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:02.622+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:02.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:02.622+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:02.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:02.622+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:02.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:02.622+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:02.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:02.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:02.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:02.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:02.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:02.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:02.595+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:03.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:03.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:03.670 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:03.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:03.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:03.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:03.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:03.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:03.671 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:03.612+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:03.671 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:03.612+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:03.672 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:03.612+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:03.672 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:03.613+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:03.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:03.597+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:03.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:03.597+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:03.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:03.597+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:03.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:03.597+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:04.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:04.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:04.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:04.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:04.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:04.383+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:04.563+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:04.563+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:04.563+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:04.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:04.563+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:04.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:04.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:04.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:04.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:04.569+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:04.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:04.569+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:05.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:05.359+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:05.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:05.359+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:05.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:05.359+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:05.516+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:05.516+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:05.516+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:05.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:05.516+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:05.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:05.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:05.587+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:05.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:05.587+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:06.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:06.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:06.372+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:06.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:06.372+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:06.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:06.372+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:06.558+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:06.558+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:06.558+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:06.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:06.558+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:06.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:06.540+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:06.540+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:06.540+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:06.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:06.540+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:07.400 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:07.052+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10011 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:07.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:07.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:07.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:07.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:07.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:07.414+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:07.601+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:07.601+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:07.601+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:07.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:07.601+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:07.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:07.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:07.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:07.549+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:07.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:07.549+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:08.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:08.441+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:08.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:08.441+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:08.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:08.441+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:08.591+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:08.591+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:08.591+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:08.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:08.591+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:08.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:08.562+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:08.562+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:08.562+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:08.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:08.563+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:09.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:09.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:09.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:09.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:09.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:09.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:09.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:09.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:09.570+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:09.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:09.570+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:09.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:09.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:09.588+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:09.708 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:09.588+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:10.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:10.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:10.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:10.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:10.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:10.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:10.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:10.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:10.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:10.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:10.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:10.638+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:10.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:10.638+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:10.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:10.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:10.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:10.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:10.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:10.595+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:10.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:10.595+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:11.625 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:11.388+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:11.625 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:11.388+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:11.625 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:11.388+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:11.625 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:11.617+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:11.626 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:11.617+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:11.626 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:11.617+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:11.626 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:11.617+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:11.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:11.638+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:11.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:11.638+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:11.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:11.638+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:11.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:11.638+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:12.399 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:12.053+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10063 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:12.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:12.413+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:12.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:12.413+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:12.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:12.413+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:12.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:12.609+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:12.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:12.609+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:12.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:12.609+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:12.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:12.609+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:12.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:12.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:12.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:12.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:12.642+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:12.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:12.642+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:13.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:13.454+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:13.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:13.454+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:13.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:13.454+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:13.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:13.651+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:13.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:13.651+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:13.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:13.651+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:13.707 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:13.652+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:13.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:13.607+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:13.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:13.607+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:13.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:13.607+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:13.708 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:13.607+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:14.683 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:14.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:14.683 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:14.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:14.683 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:14.442+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:14.683 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:14.560+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:14.684 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:14.560+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:14.684 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:14.560+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:14.684 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:14.560+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:14.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:14.697+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:14.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:14.697+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:14.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:14.697+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:14.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:14.697+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:15.672 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:15.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:15.672 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:15.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:15.673 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:15.394+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:15.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:15.551+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:15.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:15.551+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:15.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:15.551+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:15.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:15.552+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:15.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:15.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:15.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:15.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:15.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:15.686+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:15.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:15.686+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:16.680 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:16.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:16.680 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:16.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:16.681 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:16.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:16.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:16.546+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:16.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:16.546+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:16.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:16.546+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:16.681 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:16.546+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:16.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:16.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:16.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:16.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:16.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:16.694+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:16.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:16.694+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:17.374 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:17.054+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10114 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:17.670 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:17.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:17.670 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:17.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:17.670 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:17.507+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:17.670 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:17.507+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:17.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:17.388+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:17.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:17.388+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:17.671 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:17.388+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:17.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:17.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:17.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:17.684+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:17.684+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:18.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:18.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:18.693 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:18.426+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:18.693 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:18.426+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:18.694 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:18.426+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:18.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:18.524+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:18.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:18.524+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:18.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:18.524+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:18.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:18.524+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:18.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:18.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:18.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:18.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:18.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:18.707+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:18.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:18.707+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:19.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:19.486+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:19.702 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:19.486+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:19.702 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:19.486+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:19.702 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:19.486+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:19.702 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:19.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:19.702 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:19.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:19.703 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:19.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:19.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:19.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:19.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:19.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:19.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:19.715+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:19.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:19.715+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:20.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:20.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:20.496+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:20.496+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:20.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:20.445+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:20.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:20.445+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:20.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:20.445+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:20.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:20.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:20.758+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:20.758+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:21.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:21.445+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:21.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:21.445+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:21.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:21.445+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:21.465+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:21.465+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:21.465+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:21.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:21.465+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:21.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:22.042 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:21.784+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:22.043 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:21.784+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:22.043 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:21.784+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:22.043 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:21.784+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:22.445 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:22.056+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10167 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:22.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:22.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:22.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:22.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:22.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:22.459+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:22.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:22.513+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:22.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:22.513+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:22.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:22.513+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:22.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:22.513+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:23.190 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:22.803+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:23.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:22.803+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:23.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:22.803+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:23.191 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:22.803+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:23.806 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:23.501+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:23.806 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:23.501+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:23.806 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:23.501+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:23.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:23.520+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:23.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:23.520+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:23.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:23.520+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:23.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:23.520+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:24.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:23.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:24.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:23.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:24.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:23.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:24.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:23.820+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:24.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:24.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:24.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:24.830 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:18:24.831 INFO:tasks.cephadm:Teardown complete 2024-04-15T23:18:24.831 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:18:24.855 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-15T23:18:24.869 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-04-15T23:18:24.885 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-15T23:18:24.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:24.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:24.889 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:24.517+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:24.889 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:24.517+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:24.890 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:24.517+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:24.890 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:24.518+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:24.890 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:24.507+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:24.890 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:24.507+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:24.890 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:24.507+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:24.852+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:24.852+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:24.852+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:24.852+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:25.822 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:25.519+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:25.822 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:25.519+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:25.822 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:25.519+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:25.823 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:25.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:25.823 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:25.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:25.823 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:25.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:25.823 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:25.494+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:26.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:25.836+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:26.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:25.836+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:26.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:25.836+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:26.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:25.836+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:26.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:26.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:26.505+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:26.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:26.505+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:26.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:26.482+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:26.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:26.482+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:26.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:26.482+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:27.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:27.057+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10218 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:27.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:26.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:27.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:26.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:27.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:26.820+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:27.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:26.820+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:27.504 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:27.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:27.504 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:27.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:27.504 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:27.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:27.806 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:27.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:27.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:27.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:27.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:27.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:27.807 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:27.518+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:27.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:27.821+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:27.821+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:27.821+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:27.821+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:28.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:28.463+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:28.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:28.463+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:28.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:28.463+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:28.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:28.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:28.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:28.485+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:28.865+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:28.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:28.866+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:28.866+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:29.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:29.424+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:29.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:29.424+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:29.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:29.424+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:29.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:29.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:29.518+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:29.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:29.519+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:29.908+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:29.909+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:29.909+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:29.909+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:30.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:30.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:30.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:30.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:30.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:30.467+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:30.531+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:30.531+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:30.531+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:30.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:30.532+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:30.915+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:30.915+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:30.915+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:30.915+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:31.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:31.440+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:31.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:31.440+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:31.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:31.440+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:31.558+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:31.558+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:31.558+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:31.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:31.558+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:32.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:32.060+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10270 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:32.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:31.889+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:31.889+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:31.889+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:31.890+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:32.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:32.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:32.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:32.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:32.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:32.397+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:32.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:32.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:32.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:32.586+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:33.008 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:32.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:33.008 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:32.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:33.009 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:32.900+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:33.009 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:32.900+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:33.380+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:33.380+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:33.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:33.380+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:33.559+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:33.559+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:33.559+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:33.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:33.559+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:33.909+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:33.909+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:33.909+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:33.909+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:34.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:34.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:34.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:34.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:34.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:34.392+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:34.588+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:34.588+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:34.588+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:34.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:34.588+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:35.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:34.936+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:35.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:34.936+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:35.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:34.936+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:35.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:34.936+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:35.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:35.427+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:35.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:35.427+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:35.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:35.427+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:35.566+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:35.566+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:35.566+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:35.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:35.567+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:36.366 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:35.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:36.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:35.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:36.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:35.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:36.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:35.980+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:36.534+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:36.534+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:36.534+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:36.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:36.534+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:36.381+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:36.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:36.381+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:36.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:36.381+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:36.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:36.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:37.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:37.061+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10321 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:37.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:36.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:36.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:36.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:36.966+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:37.579+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:37.579+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:37.579+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:37.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:37.579+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:37.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:37.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:37.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:37.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:38.426 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:37.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:38.427 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:37.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:38.427 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:37.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:38.427 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:37.977+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:38.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:38.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:38.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:38.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:38.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:38.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:38.600+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:38.441+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:38.441+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:38.708 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:38.441+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:39.407 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:39.407 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:39.407 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:39.407 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.019+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:39.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:39.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:39.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:39.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:39.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:39.632+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:39.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:39.632+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:39.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:39.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:39.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:39.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:39.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:39.421+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:40.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:40.441 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.975+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:40.441 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.975+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:40.442 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.975+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:40.442 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:39.975+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:40.669+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:40.669+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:40.669+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:40.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:40.669+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:40.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:40.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:40.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:40.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:40.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:40.456+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:41.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:40.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:41.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:40.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:41.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:40.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:41.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:40.980+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:41.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:41.676+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:41.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:41.676+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:41.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:41.676+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:41.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:41.676+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:41.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:41.477+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:41.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:41.477+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:41.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:41.477+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:42.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:42.063+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10374 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:42.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:41.986+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:42.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:41.986+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:42.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:41.986+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:42.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:41.986+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:42.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:42.659+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:42.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:42.659+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:42.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:42.659+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:42.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:42.659+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:42.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:42.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:42.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:42.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:42.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:42.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:43.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:43.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:43.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:43.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:43.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:43.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:43.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:43.010+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:43.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:43.504+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:43.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:43.504+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:43.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:43.504+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:43.621+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:43.621+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:43.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:43.621+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:43.958 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:43.621+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:44.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:44.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:44.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:44.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:44.446 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:44.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:44.447 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:44.000+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:44.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:44.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:44.594+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:44.594+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:44.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:44.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:44.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:44.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:44.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:44.461+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:45.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:45.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:45.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:45.046+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:45.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:45.621+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:45.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:45.621+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:45.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:45.621+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:45.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:45.621+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:45.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:45.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:45.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:45.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:45.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:45.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:46.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.004+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:46.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.004+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:46.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.004+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:46.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.004+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:46.952 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:46.646+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:46.952 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:46.646+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:46.952 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:46.646+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:46.952 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:46.646+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:46.953 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:46.544+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:46.953 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:46.544+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:46.953 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:46.544+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:47.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:47.064+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10424 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:47.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:47.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:46.966+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:47.766 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:47.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:47.767 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:47.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:47.767 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:47.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:47.767 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:47.613+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:47.767 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:47.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:47.767 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:47.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:47.768 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:47.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:48.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:47.960+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:48.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:47.960+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:48.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:47.960+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:47.960+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:48.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:48.543+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:48.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:48.543+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:48.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:48.543+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:48.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:48.639+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:48.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:48.639+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:48.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:48.639+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:48.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:48.639+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:49.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:49.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:49.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:49.005+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:49.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:49.494+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:49.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:49.494+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:49.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:49.494+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:49.623+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:49.623+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:49.623+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:49.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:49.623+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:50.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:50.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:50.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:50.028+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:50.774 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:50.493+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:50.774 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:50.493+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:50.775 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:50.493+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:50.775 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:50.589+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:50.775 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:50.589+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:50.775 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:50.589+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:50.775 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:50.589+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:51.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:51.023+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:51.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:51.023+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:51.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:51.023+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:51.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:51.023+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:51.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:51.541+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:51.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:51.541+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:51.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:51.541+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:51.569+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:51.569+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:51.569+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:51.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:51.569+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:52.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:52.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:52.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:52.066+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10476 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:52.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:52.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:52.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:52.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:52.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:52.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:52.012+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:52.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:52.536+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:52.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:52.536+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:52.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:52.536+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:52.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:52.545+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:52.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:52.545+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:52.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:52.545+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:52.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:52.546+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.005+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:53.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.005+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:53.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:53.574+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:53.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:53.574+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:53.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:53.574+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:53.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:53.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:53.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:53.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:53.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:53.532+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:53.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:53.533+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:53.983+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:54.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:54.534+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:54.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:54.534+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:54.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:54.534+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:54.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:54.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:54.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:54.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:54.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:54.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:54.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:54.504+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:55.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:54.981+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:54.981+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:54.981+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:54.981+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:55.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:55.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:55.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:55.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:55.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:55.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:55.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:55.541+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:55.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:55.501+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:55.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:55.501+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:55.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:55.501+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:56.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:55.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:56.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:55.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:56.450 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:55.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:56.451 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:55.987+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:56.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:56.585+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:56.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:56.585+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:56.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:56.585+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:56.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:56.585+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:56.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:56.465+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:56.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:56.465+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:56.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:56.465+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:57.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:18:57.066+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10526 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:18:57.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:56.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:56.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:56.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:56.998+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:57.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:57.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:57.586+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:57.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:57.586+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:57.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:57.500+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:57.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:57.500+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:57.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:57.500+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:58.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:58.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:58.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:18:58.174 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:18:58.224 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:18:58.242 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T23:18:58.253 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T23:18:58.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:18:58.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:18:58.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:57.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:58.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:57.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:58.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:57.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:58.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:57.990+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:58.636+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:58.636+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:58.636+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:58.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:58.636+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:58.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:58.498+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:58.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:58.498+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:58.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:58.498+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:59.444 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:59.022+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:59.444 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:59.022+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:59.444 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:59.022+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:18:59.444 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:18:59.022+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:59.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:59.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:59.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:59.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:59.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:59.613+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:18:59.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:18:59.613+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:18:59.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:59.458+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:59.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:59.458+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:18:59.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:18:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:18:59.458+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:00.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:00.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:00.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:00.045+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:00.585+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:00.585+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:00.585+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:00.957 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:00.585+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:00.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:00.491+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:00.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:00.491+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:00.958 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:00.491+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:01.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:19:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:01.007+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:01.007+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:01.007+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:01.007+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:01.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:01.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:01.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:01.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:01.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:01.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:01.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:01.584+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:01.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:01.516+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:01.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:01.516+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:01.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:01.516+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:02.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:02.068+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10578 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:02.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:02.003+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:02.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:02.003+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:02.003+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:02.003+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:02.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:02.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:02.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:02.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:02.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:02.600+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:02.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:02.600+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:02.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:02.540+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:02.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:02.540+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:02.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:02.540+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:03.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:03.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:03.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:03.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:03.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:03.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:03.042+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:03.579+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:03.579+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:03.579+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:03.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:03.579+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:03.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:03.583+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:03.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:03.583+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:03.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:03.583+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:04.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:04.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:04.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.019+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:04.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.019+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:04.578+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:04.578+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:04.578+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:04.956 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:04.578+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:04.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:04.601+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:04.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:04.601+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:04.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:04.601+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:04.996+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:05.954 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:05.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:05.954 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:05.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:05.954 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:05.584+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:05.955 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:05.584+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:05.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:05.598+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:05.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:05.598+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:05.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:05.598+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:06.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:05.968+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:06.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:05.968+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:06.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:05.968+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:06.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:05.968+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:06.911 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:06.549+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:06.911 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:06.549+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:06.911 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:06.549+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:06.911 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:06.550+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:06.912 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:06.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:06.912 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:06.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:06.912 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:06.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:07.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:07.069+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10629 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:07.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:06.925+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:07.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:06.925+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:07.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:06.925+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:07.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:06.925+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:07.886 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:07.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:07.886 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:07.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:07.887 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:07.627+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:07.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:07.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:07.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:07.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:07.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:07.504+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:07.887 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:07.504+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:08.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:07.899+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:08.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:07.899+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:08.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:07.899+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:08.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:07.900+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:08.643 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:08.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:08.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:08.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:08.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:08.541+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:08.644 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:08.541+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:08.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:08.857+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:08.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:08.857+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:08.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:08.857+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:08.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:08.857+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:08.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:08.657+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:08.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:08.657+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:08.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:08.657+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:09.663 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:09.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:09.664 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:09.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:09.664 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:09.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:09.664 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:09.494+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:09.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:09.817+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:09.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:09.817+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:09.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:09.817+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:09.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:09.817+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:09.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:09.678+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:09.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:09.678+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:09.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:09.678+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:10.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:19:10.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:19:10.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:10.452+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:10.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:10.452+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:10.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:10.452+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:10.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:10.452+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:10.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:10.695+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:10.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:10.695+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:10.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:10.695+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:10.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:10.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:10.855+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:10.855+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:11.855 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:11.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:11.855 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:11.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:11.856 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:11.492+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:11.856 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:11.492+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:11.856 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:11.701+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:11.857 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:11.701+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:11.857 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:11.701+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:12.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:12.071+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10681 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:12.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:11.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:12.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:11.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:12.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:11.869+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:12.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:11.869+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:12.853 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:12.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:12.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:12.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:12.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:12.485+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:12.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:12.485+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:12.854 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:12.685+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:12.854 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:12.685+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:12.854 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:12.685+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:12.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:12.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:12.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:12.867+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:13.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:19:13.634 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:13.460+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:13.634 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:13.460+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:13.635 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:13.460+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:13.635 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:13.460+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:13.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:13.884+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:13.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:13.884+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:13.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:13.884+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:13.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:13.884+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:13.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:13.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:13.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:13.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:13.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:13.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:14.833 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:14.499+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:14.833 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:14.499+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:14.833 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:14.499+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:14.834 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:14.499+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:14.834 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:14.687+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:14.834 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:14.687+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:14.834 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:14.687+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:14.847+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:14.847+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:14.847+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:14.847+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:15.853 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:15.495+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:15.853 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:15.495+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:15.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:15.495+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:15.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:15.495+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:15.854 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:15.712+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:15.854 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:15.712+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:15.854 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:15.712+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:15.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:15.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:15.867+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:15.867+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:16.452+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:16.452+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:16.452+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:16.452+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:16.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:16.680+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:16.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:16.680+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:16.707 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:16.680+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:17.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:17.072+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10732 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:17.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:16.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:17.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:16.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:17.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:16.823+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:17.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:16.823+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:17.696 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:17.431+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:17.697 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:17.431+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:17.697 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:17.431+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:17.697 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:17.431+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:17.711+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:17.711+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:17.711+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:17.844+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:17.844+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:17.844+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:17.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:17.844+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:18.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:18.387+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:18.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:18.387+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:18.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:18.387+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:18.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:18.387+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:18.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:18.686+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:18.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:18.686+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:18.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:18.686+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:18.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:18.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:18.837+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:18.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:18.837+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:19.427+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:19.427+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:19.427+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:19.427+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:20.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:19.725+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:20.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:19.725+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:20.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:19.725+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:20.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:19.845+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:20.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:19.845+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:20.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:19.845+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:20.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:19.845+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:20.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:20.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:20.380+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:20.380+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:20.982 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:20.822+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:20.982 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:20.822+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:20.983 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:20.822+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:20.983 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:20.823+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:20.983 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:20.728+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:20.983 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:20.728+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:20.983 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:20.728+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:21.676 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:21.428+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:21.677 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:21.428+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:21.677 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:21.428+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:21.677 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:21.428+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:21.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:21.691+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:21.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:21.691+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:21.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:21.691+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:21.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:21.868+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:21.868+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:21.868+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:21.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:21.868+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:22.383 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:22.075+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10785 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:22.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:22.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:22.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:22.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:22.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:22.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:22.673 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:22.398+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:22.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:22.687+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:22.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:22.687+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:22.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:22.687+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:22.907+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:22.907+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:22.907+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:22.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:22.907+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:23.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:23.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:23.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:23.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:23.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:23.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:23.672 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:23.378+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:23.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:23.686+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:23.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:23.686+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:23.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:23.686+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:23.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:23.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:23.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:23.957 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:23.927+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:24.700 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:24.338+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:24.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:24.338+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:24.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:24.338+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:24.701 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:24.338+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:24.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:24.930+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:24.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:24.930+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:24.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:24.930+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:24.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:24.930+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:24.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:24.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:24.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:24.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:24.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:24.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:25.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:19:25.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:19:25.691 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:25.323+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:25.691 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:25.323+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:25.692 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:25.323+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:25.692 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:25.323+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:25.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:25.954+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:25.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:25.954+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:25.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:25.954+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:25.956 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:25.954+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:25.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:25.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:25.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:25.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:25.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:25.706+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:26.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:26.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:26.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:26.368+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:27.062 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:26.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:27.062 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:26.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:27.063 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:26.751+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:27.063 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:26.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:27.063 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:26.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:27.063 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:26.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:27.063 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:26.983+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:27.343 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:27.076+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10836 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:27.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:27.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:27.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:27.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:27.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:27.358+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:27.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:27.358+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:28.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:27.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:28.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:27.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:28.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:27.760+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:27.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:28.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:27.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:28.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:27.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:28.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:27.987+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:28.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:19:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:28.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:28.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:28.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:28.396+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:29.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:28.742+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:29.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:28.742+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:29.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:28.742+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:28.941+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:29.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:28.941+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:29.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:28.941+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:29.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:28.941+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:29.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:29.379+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:29.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:29.379+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:29.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:29.379+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:29.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:29.379+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:30.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:29.774+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:30.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:29.774+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:30.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:29.774+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:29.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:29.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:29.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:29.987+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:30.416+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:30.416+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:30.416+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:30.416+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:31.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:30.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:31.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:30.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:31.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:30.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:30.968+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:30.968+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:31.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:30.968+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:31.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:30.968+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:31.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:19:31.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:19:31.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:19:31.518 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:19:31.519 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:19:31.534 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T23:19:31.545 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T23:19:31.546 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T23:19:31.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:19:31.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:19:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:31.419+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:31.419+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:31.419+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:31.419+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:32.064 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:31.758+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:32.064 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:31.758+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:32.064 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:31.758+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:32.065 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:32.003+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:32.065 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:32.003+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:32.065 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:32.003+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:32.065 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:32.003+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:32.427 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:32.078+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10888 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:32.442+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:32.442+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:32.442+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:32.709 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:32.442+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:33.012 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:32.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:33.012 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:32.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:33.012 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:32.741+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:33.027+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:33.027+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:33.027+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:33.382 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:33.027+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:33.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:33.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:33.396+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:33.396+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:34.020 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:33.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:34.020 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:33.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:34.020 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:33.723+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:34.406 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:34.034+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:34.406 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:34.035+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:34.406 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:34.035+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:34.406 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:34.035+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:34.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:19:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:34.420+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:34.420+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:34.420+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:34.420+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:34.768+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:34.768+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:34.768+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:35.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:35.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:35.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:35.018+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:35.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:35.439+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:35.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:35.439+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:35.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:35.439+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:35.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:35.439+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:36.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:35.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:36.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:35.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:36.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:35.756+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:36.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:36.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:36.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:36.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:36.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:36.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:36.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:36.000+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:36.413+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:36.413+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:36.413+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:36.413+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:37.065 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:37.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:37.066 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:37.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:37.066 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:37.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:37.066 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:37.010+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:37.066 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:36.762+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:37.067 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:36.762+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:37.067 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:36.762+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:37.414 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:37.080+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10939 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:37.429+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:37.429+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:37.429+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:37.429+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:38.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:37.778+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:38.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:37.778+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:38.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:37.778+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:38.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:38.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:38.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:38.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.000+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:38.774 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:38.474+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:38.774 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:38.474+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:38.774 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:38.474+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:38.774 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:38.474+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:38.788+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:38.788+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:38.788+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:38.987+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:39.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:39.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:39.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:39.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:39.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:39.494+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:39.813 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:39.494+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:40.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:39.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:40.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:39.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:40.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:39.828+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.011+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.011+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.012+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:40.845 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:40.514+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:40.845 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:40.514+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:40.845 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:40.514+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:40.845 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:40.514+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:40.860+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:40.860+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:40.860+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:41.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.984+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:41.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.984+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:41.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.984+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:41.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:40.985+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:41.853 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:41.515+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:41.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:41.515+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:41.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:41.515+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:41.854 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:41.515+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:42.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:42.081+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 10992 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:42.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:41.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:42.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:41.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:42.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:41.868+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:42.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:41.993+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:42.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:41.993+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:42.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:41.993+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:42.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:41.993+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:42.468+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:42.468+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:42.468+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:42.468+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:43.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:42.845+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:43.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:42.845+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:43.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:42.845+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:43.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:42.993+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:43.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:42.993+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:43.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:42.993+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:43.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:42.993+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:43.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:19:43.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:19:43.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:43.455+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:43.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:43.455+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:43.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:43.455+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:43.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:43.455+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:43.869+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:43.869+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:43.869+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:44.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:44.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:44.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:44.009+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:44.408+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:44.408+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:44.408+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:44.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:44.408+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:45.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:44.847+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:45.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:44.847+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:45.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:44.847+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:45.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:45.024+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:45.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:45.024+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:45.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:45.024+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:45.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:45.024+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:45.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:45.400+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:45.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:45.400+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:45.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:45.400+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:45.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:45.400+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:46.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:45.863+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:46.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:45.863+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:46.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:45.863+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:46.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:46.072+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:46.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:46.072+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:46.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:46.072+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:46.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:46.073+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:46.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:19:46.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:46.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:46.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:46.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:46.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:46.352+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:46.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:46.352+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:47.098 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:47.082+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11043 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:47.098 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:46.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:47.099 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:46.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:47.099 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:46.850+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:47.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:47.331+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:47.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:47.331+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:47.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:47.331+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:47.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:47.331+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:47.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:47.112+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:47.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:47.112+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:47.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:47.112+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:47.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:47.112+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:48.069 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:47.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:48.069 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:47.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:48.069 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:47.832+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:48.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:48.322+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:48.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:48.322+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:48.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:48.322+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:48.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:48.322+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:48.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:48.083+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:48.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:48.083+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:48.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:48.083+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:48.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:48.083+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:48.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:48.875+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:48.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:48.875+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:48.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:48.875+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:49.305 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:49.061+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:49.305 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:49.061+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:49.305 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:49.061+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:49.305 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:49.061+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:49.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:49.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:49.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:49.319+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:50.077 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:49.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:50.077 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:49.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:50.078 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:49.840+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:50.350 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:50.091+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:50.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:50.091+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:50.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:50.091+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:50.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:50.091+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:50.364+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:50.364+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:50.364+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:50.364+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:51.086 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:50.806+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:51.087 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:50.806+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:51.087 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:50.806+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:51.336 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:51.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:51.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:51.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:51.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:51.100+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:51.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:51.100+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:51.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:51.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:51.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:51.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:51.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:51.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:51.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:51.351+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:52.069 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:51.770+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:52.069 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:51.770+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:52.069 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:51.770+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:52.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:52.083+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11095 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:52.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:52.121+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:52.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:52.121+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:52.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:52.121+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:52.121+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:52.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:52.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:52.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:52.309+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:53.153 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:52.737+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:53.154 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:52.737+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:53.154 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:52.737+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:53.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:53.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:53.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:53.168+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:53.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:53.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:53.279+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:53.279+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:53.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:53.735+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:53.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:53.735+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:53.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:53.735+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:54.158+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:54.158+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:54.158+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:54.158+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:54.305+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:54.305+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:54.305+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:54.305+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:54.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:54.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:54.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:54.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:54.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:54.715+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:55.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:55.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:55.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:55.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:55.181+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:55.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:55.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:55.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:55.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:55.261+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:55.261+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:55.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:55.667+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:55.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:55.667+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:55.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:55.667+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:56.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:56.307+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:56.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:56.307+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:56.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:56.307+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:56.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:56.307+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:56.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:56.143+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:56.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:56.143+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:56.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:56.143+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:56.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:56.143+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:56.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:56.682+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:56.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:56.682+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:56.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:56.682+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:57.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:19:57.085+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11146 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:19:57.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:57.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:57.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:57.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:57.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:57.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:57.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:57.294+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:57.131+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:57.131+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:57.131+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:57.131+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:57.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:57.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:57.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:57.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:57.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:57.633+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:58.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:58.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:58.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:58.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:58.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:58.262+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:58.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:58.262+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:58.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:58.114+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:58.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:58.114+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:58.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:58.114+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:58.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:58.114+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:58.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:19:58.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:19:58.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:58.589+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:58.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:58.589+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:58.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:58.589+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:59.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:59.218+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:59.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:59.218+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:59.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:59.218+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:19:59.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:19:59.218+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:59.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:59.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:59.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:59.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:59.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:59.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:19:59.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:19:59.107+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:19:59.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:59.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:59.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:59.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:19:59.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:19:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:19:59.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:00.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:00.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:00.115+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:00.115+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:00.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:00.187+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:00.187+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:00.187+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:00.187+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:00.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:00.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:00.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:00.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:00.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:00.648+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:01.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:01.168+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:01.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:01.168+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:01.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:01.168+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:01.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:01.168+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:01.159+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:01.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:01.159+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:01.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:01.159+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:01.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:01.159+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:01.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:01.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:01.649+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:01.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:01.649+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:01.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:01.649+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:02.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:02.086+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11199 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:02.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:02.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:02.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:02.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:02.157+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:02.197+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:02.197+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:02.197+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:02.197+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:02.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:02.637+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:02.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:02.637+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:02.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:02.637+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:03.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:03.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:03.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:03.132+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:03.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:03.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:03.200+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:03.200+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:03.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:03.659+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:03.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:03.659+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:03.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:03.659+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:04.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:04.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:04.169+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:04.169+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:04.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:04.160+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:04.160+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:04.160+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:04.160+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:04.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:04.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:04.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:04.830 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:20:04.831 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:20:04.840 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T23:20:04.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T23:20:04.889 INFO:teuthology.task.internal:Duration was 2252.019252 seconds 2024-04-15T23:20:04.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T23:20:04.931 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T23:20:04.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:04.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:04.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:04.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:04.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:04.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:04.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:04.619+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:05.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:05.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:05.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:05.146+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:05.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:05.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:05.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:05.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:05.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:05.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:05.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:05.123+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:05.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:05.595+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:05.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:05.595+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:05.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:05.595+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:06.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:06.137+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:06.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:06.137+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:06.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:06.137+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:06.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:06.137+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:06.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:06.122+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:06.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:06.122+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:06.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:06.122+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:06.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:06.122+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:06.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:06.624+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:06.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:06.624+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:06.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:06.624+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:07.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:07.088+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11250 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:07.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:07.107+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:07.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:07.107+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:07.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:07.107+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:07.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:07.107+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:07.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:07.097+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:07.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:07.097+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:07.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:07.097+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:07.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:07.097+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:07.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:07.578+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:07.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:07.578+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:07.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:07.578+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:07.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:08.097+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:08.097+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:08.097+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:08.097+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:08.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:08.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:08.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:08.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:08.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:08.145+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:08.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:08.145+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:08.799 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:08.556+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:08.799 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:08.556+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:08.799 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:08.556+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:09.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:09.134+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:09.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:09.134+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:09.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:09.134+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:09.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:09.134+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:09.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:09.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:09.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:09.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:09.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:09.123+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:09.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:09.124+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:09.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:09.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:09.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:09.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:09.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:09.605+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:10.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:10.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:10.094+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:10.095+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:10.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:10.078+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:10.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:10.078+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:10.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:10.078+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:10.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:10.078+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:10.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:10.596+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:10.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:10.596+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:10.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:10.596+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:11.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:11.106+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:11.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:11.106+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:11.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:11.106+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:11.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:11.106+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:11.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:11.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:11.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:11.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:11.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:11.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:11.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:11.043+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:11.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:11.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:11.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:11.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:11.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:11.599+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:12.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:12.090+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11302 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:12.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:12.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:12.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:12.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:12.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:12.064+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:12.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:12.064+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:12.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:12.088+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:12.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:12.088+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:12.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:12.088+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:12.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:12.088+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:12.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:12.553+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:12.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:12.553+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:12.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:12.553+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:13.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:13.075+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:13.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:13.075+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:13.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:13.075+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:13.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:13.075+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:13.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:13.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:13.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:13.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:13.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:13.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:13.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:13.041+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:13.840 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:13.521+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:13.840 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:13.521+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:13.841 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:13.521+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:14.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:14.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:14.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:14.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:14.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:14.032+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:14.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:14.032+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:14.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.044+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:14.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.044+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:14.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.044+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:14.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.044+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:14.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:14.535+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:14.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:14.535+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:14.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:14.535+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:15.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:15.017+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:15.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:15.017+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:15.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:15.017+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:15.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:15.017+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:15.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.999+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:15.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.999+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:15.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.999+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:15.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:14.999+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:15.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:15.554+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:15.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:15.554+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:15.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:15.554+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:16.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:16.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:16.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:16.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.030+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:16.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.013+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:16.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:16.552+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:16.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:16.552+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:16.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:16.552+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:16.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:17.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:17.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:17.092+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11353 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:17.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:16.997+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.990+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:16.990+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:17.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:17.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:17.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:17.988+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:17.988+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:17.988+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:17.988+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:18.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:18.030+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:18.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:18.030+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:18.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:18.030+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:18.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:18.030+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:18.838 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:18.522+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:18.838 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:18.523+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:18.838 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:18.523+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:19.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:18.993+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:19.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:18.993+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:19.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:18.993+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:19.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:18.993+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:19.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:19.060+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:19.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:19.060+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:19.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:19.060+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:19.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:19.060+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:19.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:19.511+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:19.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:19.511+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:19.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:19.511+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:20.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:20.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:20.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:20.016+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:20.016+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:20.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:20.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:20.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:20.077+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:20.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:20.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:20.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:20.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:20.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:20.547+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:21.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.013+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:21.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.013+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:21.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.013+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:21.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.013+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:21.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:21.047+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:21.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:21.047+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:21.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:21.047+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:21.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:21.047+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:21.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:21.535+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:21.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:21.535+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:21.957 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:21.535+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:22.456 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:22.093+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11406 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:22.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:22.081+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:22.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:22.081+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:22.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:22.081+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:22.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:22.081+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:22.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.974+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:22.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.974+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:22.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.974+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:22.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:21.974+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:22.934 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:22.584+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:22.934 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:22.584+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:22.934 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:22.584+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:23.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:23.065+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:23.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:23.065+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:23.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:23.065+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:23.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:23.065+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:23.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:22.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:23.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:22.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:23.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:22.997+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:23.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:22.997+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:23.922 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:23.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:23.923 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:23.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:23.923 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:23.573+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:24.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:23.958+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:24.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:23.958+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:24.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:23.958+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:24.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:23.958+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:24.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:24.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:24.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:24.028+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:24.028+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:24.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:24.530+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:24.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:24.530+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:24.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:24.530+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:25.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:24.992+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:25.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:24.992+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:25.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:24.992+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:25.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:24.992+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:25.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:25.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:25.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:25.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:25.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:25.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:09:53.184227+0000 (oldest deadline 2024-04-15T23:10:13.184227+0000) 2024-04-15T23:20:25.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:25.019+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:25.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:25.534+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:25.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:25.534+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:25.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:25.534+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:26.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:25.990+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:26.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:26.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:26.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:26.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:26.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:26.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:26.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:26.021+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:26.934 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:26.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:26.935 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:26.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:26.935 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:26.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:27.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:27.095+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11457 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:27.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:26.949+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:27.023+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:27.023+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:27.023+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:27.023+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:27.906 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:27.496+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:27.907 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:27.496+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:27.907 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:27.496+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:28.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:27.921+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:28.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:28.012+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:28.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:28.012+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:28.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:28.012+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:28.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:28.012+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:28.928 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:28.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:28.929 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:28.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:28.929 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:28.508+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:28.943+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:29.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:29.049+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:29.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:29.049+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:29.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:29.049+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:29.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:29.049+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:29.929 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:29.486+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:29.929 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:29.486+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:29.929 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:29.486+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:29.944+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:30.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:30.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:30.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:30.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:30.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:30.088+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:30.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:30.088+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:30.943 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:30.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:30.943 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:30.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:30.943 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:30.525+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:30.957+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:31.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:31.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:31.093+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:31.093+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:31.924 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:31.488+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:31.924 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:31.488+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:31.924 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:31.488+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:32.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:32.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:32.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:32.097+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11509 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:32.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:31.939+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:32.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:32.053+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:32.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:32.053+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:32.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:32.053+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:32.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:32.053+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:32.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:32.440+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:32.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:32.440+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:32.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:32.440+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:33.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:32.914+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:33.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:33.054+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:33.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:33.054+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:33.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:33.054+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:33.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:33.054+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:33.468+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:33.468+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:33.468+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:00.971937+0000 (oldest deadline 2024-04-15T23:10:20.971937+0000) 2024-04-15T23:20:34.198 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:33.957+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:34.198 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:34.091+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:34.198 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:34.091+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:34.199 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:34.091+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:34.199 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:34.091+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:35.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:35.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:34.921+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:35.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:35.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:35.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:35.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:35.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:35.058+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:35.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:35.058+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:36.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:35.924+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:36.010+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:36.010+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:36.010+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:36.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:36.010+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:37.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:37.097+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11560 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:37.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:36.931+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:37.012+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:37.012+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:37.012+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:37.012+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:38.013 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:37.948+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:38.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:38.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:38.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:38.209 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:20:38.210 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:20:38.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T23:20:38.239 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T23:20:38.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:38.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:38.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.027+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:38.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.027+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:38.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.027+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.027+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:39.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:38.953+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:39.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:39.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:39.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.982+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:39.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:38.982+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:40.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:39.961+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:40.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:40.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:40.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:40.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:40.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:40.015+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:40.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:40.015+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:41.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:40.942+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:41.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:41.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:41.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:41.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:41.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:41.021+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:10:09.796444+0000 (oldest deadline 2024-04-15T23:10:29.796444+0000) 2024-04-15T23:20:41.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:41.021+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:41.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:42.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:41.933+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:42.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:42.099+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11613 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:42.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:42.057+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:43.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:43.029+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:43.414 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:42.974+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:44.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:43.994+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:44.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:43.986+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:45.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:45.016+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:45.027+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:46.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:46.057+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:46.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:46.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:46.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:46.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:46.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:46.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:46.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:46.063+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:47.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:47.100+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11664 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:47.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:47.036+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:47.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:47.111+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:47.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:47.111+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:47.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:47.111+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:47.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:47.111+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:48.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:48.048+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:48.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:48.138+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:48.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:48.138+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:48.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:48.138+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:48.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:48.138+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:49.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:49.025+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:49.161+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:49.161+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:49.161+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:49.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:49.161+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:50.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:20:50.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:20:50.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:50.005+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:50.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:50.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:50.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:50.209+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:50.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:50.209+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:51.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:50.987+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:51.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:51.176+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:51.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:51.176+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:51.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:51.176+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:51.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:51.176+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:52.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:52.102+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11716 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:52.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:52.000+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:52.126+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:52.126+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:52.126+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:52.126+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:53.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:20:53.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:53.002+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:53.117+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:53.117+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:53.117+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:53.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:53.117+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:54.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:53.996+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:54.069+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:54.069+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:54.069+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:54.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:54.069+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:55.402 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:55.045+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:55.403 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:55.101+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:55.403 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:55.101+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:55.403 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:55.101+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:55.403 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:55.101+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:55.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:55.417+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:55.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:55.417+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:55.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:55.417+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:56.417 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:56.071+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:56.417 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:56.067+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:56.417 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:56.067+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:56.417 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:56.067+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:56.417 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:56.067+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:56.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:56.431+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:56.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:56.431+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:56.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:56.431+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:57.391 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:20:57.104+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11767 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:20:57.391 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:57.069+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:57.392 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:57.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:57.392 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:57.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:57.392 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:57.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:57.392 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:57.025+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:57.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:57.405+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:57.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:57.405+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:57.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:57.405+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:58.389 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:58.044+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:58.390 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:58.065+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:58.390 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:58.065+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:58.390 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:58.065+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:58.390 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:58.065+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:58.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:58.404+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:58.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:58.404+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:58.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:58.404+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:59.400 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:20:59.091+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:59.400 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:59.071+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:59.401 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:59.071+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:59.401 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:59.071+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:20:59.401 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:20:59.071+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:20:59.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:59.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:59.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:59.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:20:59.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:20:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:20:59.415+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:00.419 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:00.121+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:00.419 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:00.055+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:00.419 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:00.055+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:00.419 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:00.055+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:00.420 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:00.056+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:00.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:00.433+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:00.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:00.433+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:00.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:00.433+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:01.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:01.136+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:01.033+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:01.033+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:01.033+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:01.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:01.033+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:01.846 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:01.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:01.846 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:01.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:01.846 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:01.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:02.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:02.106+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11820 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:02.131+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:02.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:02.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:02.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:02.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:02.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:02.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:02.057+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:02.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:02.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:02.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:02.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:02.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:02.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:03.179+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:03.179+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:03.179+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:03.179+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:03.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:03.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:03.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:03.041+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:03.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:03.041+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:03.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:03.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:03.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:03.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:03.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:03.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:04.167+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:04.167+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:04.167+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:04.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:04.167+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:04.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:04.053+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:04.053+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:04.053+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:04.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:04.054+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:04.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:04.473+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:04.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:04.473+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:04.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:04.473+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:05.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:05.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:05.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:05.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:05.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:05.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:05.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:05.131+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:05.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:05.131+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:05.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:05.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:05.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:05.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:05.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:05.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:05.411 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:05.052+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:05.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:05.424+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:05.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:05.424+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:05.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:05.424+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:06.369 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:06.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:06.369 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:06.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:06.369 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:06.164+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:06.370 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:06.164+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:06.370 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:06.038+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:06.370 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:06.038+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:06.370 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:06.038+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:06.370 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:06.038+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:06.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:06.384+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:06.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:06.384+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:06.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:06.384+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:07.409 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:07.108+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11871 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:07.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:07.143+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:07.409 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:07.143+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:07.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:07.143+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:07.410 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:07.143+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:07.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:07.074+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:07.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:07.074+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:07.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:07.074+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:07.410 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:07.074+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:07.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:07.423+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:07.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:07.423+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:07.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:07.423+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:08.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:08.173+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:08.173+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:08.173+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:08.173+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:08.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:08.051+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:08.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:08.051+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:08.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:08.051+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:08.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:08.051+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:08.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:08.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:08.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:08.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:08.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:08.470+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:09.294 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:09.208+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:09.295 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:09.208+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:09.295 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:09.208+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:09.295 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:09.208+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:09.295 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:09.036+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:09.295 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:09.036+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:09.296 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:09.036+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:09.296 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:09.036+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:09.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:09.444+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:09.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:09.444+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:09.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:09.444+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:10.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:10.158+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:10.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:10.158+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:10.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:10.158+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:10.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:10.159+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:10.434 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:10.022+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:10.435 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:10.022+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:10.435 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:10.022+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:10.435 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:10.022+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:10.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:10.448+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:10.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:10.448+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:10.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:10.448+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:11.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:11.150+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:11.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:11.150+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:11.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:11.150+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:11.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:11.150+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:11.434 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:11.435 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:11.435 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:11.435 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.031+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:11.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:11.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:11.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:11.518 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:21:11.519 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:21:11.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T23:21:11.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:11.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:11.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:11.448+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:11.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:11.448+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:11.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:11.448+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:12.385 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:12.109+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11923 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:12.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:12.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:12.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:12.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:11.996+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:12.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:12.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:12.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:12.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:12.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:12.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:12.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:12.128+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:12.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:12.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:12.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:12.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:12.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:12.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:13.378 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:13.129+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:13.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:13.129+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:13.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:13.129+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:13.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:13.129+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:13.379 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:12.982+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:13.379 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:12.982+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:13.379 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:12.982+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:13.380 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:12.982+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:13.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:13.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:13.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:13.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:13.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:13.393+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:14.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:14.020+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:14.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:14.020+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:14.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:14.020+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:14.337 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:14.020+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:14.338 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:14.097+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:14.338 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:14.097+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:14.338 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:14.097+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:14.338 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:14.097+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:14.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:14.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:14.395+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:14.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:14.395+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:14.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:14.395+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:15.428 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:15.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:15.428 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:15.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:15.428 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:15.128+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:15.428 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:15.128+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:15.428 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:15.429 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:15.429 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.010+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:15.429 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.010+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:15.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:15.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:15.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:15.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:15.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:15.443+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:16.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:16.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:16.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:16.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:16.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:16.153+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:16.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:16.153+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:16.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:15.987+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:16.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:16.491+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:16.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:16.491+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:16.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:16.491+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:17.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:17.111+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 11975 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:17.109+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:17.109+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:17.109+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:17.109+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:17.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:16.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:16.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:16.973+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:17.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:16.973+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:17.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:17.513+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:17.955 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:17.513+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:17.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:17.513+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:18.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:18.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:18.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:18.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:18.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:18.139+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:18.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:17.975+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:18.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:17.975+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:18.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:17.975+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:18.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:17.975+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:18.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:18.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:18.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:18.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:18.956 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:18.497+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:19.378 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:18.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:19.378 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:18.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:19.379 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:18.980+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:19.379 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:18.980+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:19.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:19.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:19.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:19.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:19.379 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:19.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:19.380 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:19.174+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:19.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:19.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:19.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:19.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:19.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:19.471+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:20.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:20.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:20.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:20.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:20.174+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:20.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:19.985+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:19.985+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:19.985+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:20.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:19.985+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:20.933 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:20.481+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:20.933 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:20.481+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:20.933 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:20.481+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:21.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:21.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:21.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:21.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:21.127+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:20.948+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:20.948+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:20.948+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:20.948+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:21.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:21.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:21.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:21.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:21.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:21.447+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:22.385 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:22.113+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12029 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:22.385 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:22.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:22.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:22.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:22.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:22.127+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:22.386 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:22.127+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:22.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:21.982+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:22.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:21.982+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:22.386 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:21.982+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:22.387 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:21.982+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:22.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:22.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:22.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:22.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:22.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:22.400+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:23.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:23.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:23.205 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:23.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:23.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:23.116+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:23.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:23.116+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:23.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:22.944+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:22.944+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:23.209 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:22.944+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:23.210 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:22.944+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:23.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:23.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:23.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:23.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:23.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:23.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:23.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:23.386+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:24.115 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:23.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:24.116 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:23.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:24.116 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:23.983+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:24.116 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:23.983+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:24.419 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:24.340+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:24.420 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:24.340+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:24.420 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:24.340+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:24.420 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:24.130+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:24.420 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:24.130+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:24.420 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:24.130+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:24.421 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:24.130+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:25.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:24.952+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:24.952+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:24.952+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:24.952+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:25.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:25.123+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:25.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:25.123+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:25.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:25.123+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:25.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:25.123+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:25.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:25.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:25.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:25.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:25.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:25.363+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:26.306 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:25.974+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:25.974+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:25.974+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:25.974+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:26.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:26.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:26.171+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:26.307 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:26.171+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:26.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:26.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:26.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:26.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:26.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:26.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:26.322+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:27.206 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:27.113+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12081 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:27.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:26.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:27.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:26.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:27.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:26.927+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:27.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:26.927+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:27.155+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:27.155+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:27.155+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:27.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:27.155+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:27.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:27.279+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:27.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:27.279+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:27.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:27.279+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:28.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:27.929+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:27.929+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:27.929+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:27.929+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:28.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:28.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:28.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:28.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:28.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:28.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:28.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:28.157+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:28.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:28.306+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:28.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:28.306+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:28.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:28.306+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:28.936+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:28.936+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:28.936+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:28.936+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:29.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:29.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:29.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:29.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:29.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:29.182+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:29.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:29.182+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:29.717 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:29.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:29.717 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:29.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:29.717 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:29.290+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:30.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:29.958+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:29.958+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:29.958+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:29.959+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:30.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:30.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:30.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:30.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:30.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:30.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:30.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:30.136+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:30.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:30.306+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:30.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:30.306+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:30.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:30.306+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:31.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:30.964+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:30.964+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:30.964+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:30.964+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:31.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:31.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:31.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:31.166+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:31.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:31.166+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:31.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:31.261+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:31.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:31.261+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:31.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:31.261+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:32.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:32.115+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12134 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:32.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:32.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:32.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:32.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:32.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:32.214+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:32.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:31.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:32.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:31.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:32.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:31.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:32.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:31.970+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:32.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:32.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:32.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:32.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:32.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:32.146+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:32.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:32.146+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:33.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:32.943+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:33.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:32.943+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:33.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:32.943+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:33.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:32.943+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:33.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:33.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:33.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:33.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:33.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:33.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:33.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:33.185+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:33.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:33.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:33.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:33.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:33.242+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:34.219 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:34.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:34.220 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:34.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:34.220 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:34.209+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:34.220 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:34.209+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:34.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:33.978+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:34.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:33.978+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:34.221 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:33.978+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:34.221 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:33.978+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:34.503 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:34.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:34.504 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:34.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:34.504 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:34.234+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:35.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:35.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:35.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:35.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:35.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:35.000+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:35.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:35.000+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:35.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:35.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:35.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:35.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:35.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:35.219+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:35.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:35.219+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:35.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:35.263+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:35.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:35.263+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:35.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:35.263+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:36.249 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:36.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:36.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:36.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:36.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:36.201+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:36.250 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:36.201+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:36.250 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:36.250 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:36.251 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:36.251 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.009+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:36.705 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:36.264+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:36.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:36.264+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:36.706 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:36.264+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:37.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:37.117+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12186 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:37.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:37.217+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:37.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:37.217+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:37.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:37.217+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.967+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.967+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.967+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:36.967+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:37.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:37.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:37.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:37.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:37.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:37.180+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:38.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:38.220+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:38.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:38.220+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:38.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:38.220+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:38.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:38.001+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:38.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:38.001+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:38.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:38.001+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:38.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:38.001+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:38.165+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:38.165+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:38.165+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:38.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:38.165+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:38.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:38.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:39.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:39.193+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:39.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:39.193+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:39.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:39.193+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:39.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:39.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:39.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.015+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:39.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.015+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:39.154+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:39.154+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:39.154+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:39.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:39.154+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:40.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:40.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:40.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:40.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:40.146+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.969+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:40.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.969+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.969+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:40.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:39.969+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:40.155+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:40.208 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:40.155+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:40.155+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:40.209 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:40.155+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:41.107 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:40.985+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:41.108 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:40.985+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:41.108 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:40.985+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:41.108 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:40.985+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:41.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:41.139+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:41.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:41.139+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:41.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:41.139+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:41.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:41.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:41.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:41.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:41.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:41.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:41.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:41.149+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:41.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:42.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:42.118+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12240 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:42.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:42.159+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:42.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:42.159+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:42.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:42.159+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:42.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:42.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:42.141+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:42.141+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:42.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:42.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:42.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:42.009+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:42.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:42.009+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:43.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:43.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:43.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:43.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:43.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:43.131+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:43.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:43.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:43.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:43.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:43.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:43.139+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:43.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:43.139+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:43.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.011+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:43.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.011+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:43.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.011+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:43.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.011+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:44.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:44.109+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:44.109+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:44.109+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:44.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:44.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:44.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:43.966+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:44.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:44.095+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:44.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:44.095+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:44.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:44.095+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:44.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:44.095+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:44.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:44.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:44.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:44.830 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:21:44.831 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:21:44.848 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T23:21:44.862 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T23:21:44.863 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608/remote/smithi037 2024-04-15T23:21:44.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:44.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:45.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:45.119+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:45.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:45.119+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:45.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:45.119+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:44.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:44.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:44.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:45.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:44.996+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:45.052+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:45.052+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:45.052+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:45.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:45.052+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:46.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:46.149+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:46.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:46.149+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:46.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:46.149+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:45.956+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:45.956+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:45.956+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:46.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:45.956+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:46.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:46.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:46.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:46.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:46.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:46.042+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:46.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:46.042+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:47.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:47.120+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12292 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:47.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:47.115+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:47.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:47.115+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:47.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:47.115+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:47.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:47.081+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:47.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:47.081+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:47.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:47.081+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:47.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:47.081+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:47.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:46.948+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:47.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:46.948+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:47.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:46.948+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:47.208 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:46.948+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:47.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:21:48.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:48.105+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:48.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:48.105+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:48.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:48.105+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:48.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:48.051+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:48.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:48.051+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:48.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:48.051+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:48.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:48.051+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:47.933+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:47.933+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:47.933+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:47.933+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:49.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:49.085+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:49.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:49.085+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:49.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:49.085+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:49.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.022+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:49.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.022+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:49.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.022+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:49.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.022+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:49.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:48.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:49.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:48.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:49.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:48.977+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:49.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:48.977+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:50.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:50.040+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:50.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:50.040+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:50.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:50.040+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:49.950+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:49.950+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:49.950+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:50.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:49.950+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:50.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.991+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:50.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.991+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:50.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.991+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:50.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:49.991+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:51.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:51.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:51.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:51.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:51.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:51.045+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:51.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:51.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:51.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:51.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:51.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:51.030+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:51.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:51.030+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:51.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:50.997+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:51.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:50.997+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:51.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:50.997+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:51.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:50.997+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:52.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:52.123+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12345 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:52.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:52.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:52.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:52.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:52.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:52.002+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:52.075+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:52.075+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:52.075+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:52.075+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:52.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:52.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.018+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:53.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:52.999+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:53.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:52.999+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:53.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:52.999+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:53.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:53.118+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:53.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:53.118+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:53.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:53.118+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:53.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:53.118+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:53.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.978+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:53.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.978+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:53.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.978+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:53.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:52.978+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:54.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:54.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:54.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:54.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:54.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:54.030+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:54.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:54.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:54.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:54.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:54.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:54.136+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:54.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:54.136+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:54.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:54.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:54.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:54.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:54.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:54.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:54.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:54.017+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:55.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:55.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:55.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:55.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:55.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:55.057+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.014+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.014+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:55.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.014+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:55.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.014+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:55.125+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:55.125+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:55.125+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:55.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:55.125+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:56.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:56.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:56.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:56.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:56.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:56.010+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:56.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:55 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:55.970+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:56.120+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:56.120+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:56.120+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:56.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:56 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:56.120+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:56.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:21:56.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:21:57.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:21:57.124+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12397 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:21:57.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:57.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:57.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:57.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:57.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:57.004+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:57.099+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:57.099+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:57.099+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:57.099+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:57.016+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:57.016+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:57.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:57.016+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:57.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:57.016+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:58.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:57.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:58.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:57.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:58.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:57 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:57.957+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:58.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:58.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:58.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:58.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:58.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:58.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:58.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:58.063+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:58.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:58.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:58.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:58.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:58.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:58.148+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:58.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:58.148+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:59.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:58.980+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:59.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:58.980+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:59.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:58 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:58.980+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:21:59.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:59.108+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:59.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:59.108+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:59.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:59.108+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:21:59.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:21:59.108+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:59.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:59.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:59.132+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:21:59.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:21:59.132+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:21:59.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:00.120 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:59.968+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:00.120 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:59.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:00.120 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:21:59 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:21:59.969+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:00.455 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:00.135+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:00.135+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:00.135+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:00.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:00.135+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:00.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:00.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:00.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:00.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:00.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:00.157+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:00.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:00.157+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:01.455 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:00.977+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:01.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:00.977+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:01.456 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:00 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:00.977+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:01.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:01.203+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:01.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:01.203+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:01.457 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:01.203+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:01.458 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:01.203+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:01.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:01.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:01.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:01.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:01.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:01.168+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:01.458 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:01.168+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:02.195 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:02.126+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12451 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:02.196 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:01.958+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:02.196 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:01.958+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:02.196 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:01 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:01.958+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:02.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:02.146+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:02.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:02.146+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:02.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:02.146+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:02.197 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:02.146+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:02.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:02.210+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:02.210+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:02.210+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:02.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:02.210+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:03.162 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:02.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:03.163 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:02.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:03.163 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:02 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:02.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:03.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:03.151+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:03.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:03.151+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:03.163 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:03.151+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:03.164 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:03.151+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:03.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:03.177+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:03.177+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:03.177+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:03.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:03.178+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:04.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:03.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:04.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:03.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:04.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:03 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:03.960+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:04.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:04.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:04.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:04.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:04.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:04.181+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:04.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:04.181+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:04.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:04.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:04.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:04.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:04.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:04.185+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:04.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:04.185+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:05.129 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:04.912+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:05.129 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:04.912+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:05.129 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:04 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:04.912+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:05.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:05.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:05.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:05.174+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:05.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:05.174+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:05.144+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:05.144+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:05.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:05.144+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:05.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:05.144+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:06.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:05.962+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:06.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:05.962+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:06.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:05 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:05.962+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:06.206 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:06.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:06.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:06.180+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:06.180+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:06.132+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:06.132+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:06.132+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:06.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:06.132+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:07.157 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:07.128+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12503 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:07.157 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:06.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:07.158 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:06.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:07.158 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:06 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:06.921+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:07.158 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:07.118+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:07.158 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:07.118+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:07.158 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:07.118+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:07.158 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:07.118+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:07.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:07.172+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:07.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:07.172+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:07.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:07.172+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:07.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:07.172+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:08.192 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:07.905+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:08.193 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:07.905+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:08.193 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:07 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:07.905+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:08.193 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:08.109+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:08.193 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:08.109+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:08.193 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:08.109+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:08.194 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:08.109+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:08.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:08.207+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:08.207+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:08.207+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:08.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:08.208+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:09.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:08.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:09.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:08.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:09.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:08 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:08.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:09.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:09.079+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:09.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:09.079+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:09.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:09.079+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:09.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:09.079+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:09.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:09.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:09.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:09.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:09.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:09.239+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:09.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:09.239+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:10.093 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:09.901+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:10.094 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:09.901+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:10.094 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:09 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:09.901+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:10.455 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:10.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:10.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:10.238+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:10.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:10.238+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:10.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:10.108+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:10.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:10.108+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:10.456 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:10.108+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:10.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:10.108+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:11.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:10.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:11.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:10.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:11.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:10 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:10.939+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:11.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:11.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:11.107+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:11.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:11.107+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:11.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:11.281+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:11.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:11.281+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:11.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:11.281+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:11.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:11.281+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:12.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:22:12.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:22:12.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:12.129+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12556 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:12.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:11.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:12.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:11.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:12.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:11 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:11.891+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:12.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:12.076+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:12.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:12.076+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:12.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:12.076+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:12.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:12.076+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:12.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:12.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:12.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:12.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:12.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:12.309+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:12.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:12.309+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:13.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:12.863+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:13.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:12.863+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:13.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:12 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:12.863+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:13.033+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:13.033+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:13.033+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:13.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:13.033+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:13.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:13.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:13.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:13.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:13.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:13.267+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:13.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:13.267+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:14.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:13.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:14.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:13.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:14.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:13 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:13.893+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:14.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:14.068+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:14.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:14.068+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:14.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:14.068+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:14.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:14.068+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:14.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:14.246+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:14.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:14.246+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:14.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:14.246+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:14.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:14.246+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:15.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:15.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:14.883+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:15.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:14.883+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:15.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:14 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:14.883+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:15.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:15.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:15.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:15.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:15.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:15.063+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:15.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:15.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:15.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:15.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:15.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:15.274+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:15.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:15.274+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:16.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:15.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:16.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:15.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:16.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:15 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:15.844+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:16.037+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:16.037+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:16.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:16.037+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:16.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:16.038+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:16.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:16.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:16.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:16.294+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:16.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:16.294+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:17.117 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.016+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:17.117 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.016+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:17.117 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.016+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:17.117 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.016+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:17.117 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:16.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:17.118 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:16.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:17.118 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:16 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:16.857+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:17.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:17.130+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12608 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:17.317+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:17.317+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:17.317+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:17.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:17.317+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:18.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:22:18.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:22:18.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:18.142 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:22:18.143 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:22:18.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T23:22:18.173 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T23:22:18.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T23:22:18.211 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T23:22:18.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:22:18.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:22:18.219 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:18.219 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:18.219 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.998+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:18.219 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:17.998+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:18.220 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:17.811+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:18.220 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:17.811+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:18.220 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:17 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:17.811+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:18.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:18.328+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:18.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:18.328+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:18.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:18.328+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:18.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:18.328+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:19.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:18.955+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:19.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:18.955+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:19.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:18.955+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:19.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:18.955+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:19.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:18.808+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:18.808+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:19.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:18 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:18.808+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:19.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:19.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:19.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:19.367+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:19.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:19.367+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:20.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:19.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:20.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:19.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:20.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:19.987+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:20.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:19.987+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:20.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:19.817+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:20.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:19.817+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:20.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:19 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:19.817+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:20.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:20.360+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:20.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:20.360+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:20.360+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:20.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:20.360+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:21.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:20.855+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:21.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:20.855+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:21.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:20.855+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:20.979+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:21.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:20.979+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:21.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:20.979+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:21.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:20 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:20.979+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:21.282 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:21.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:21.370+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:21.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:21.370+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:21.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:21.370+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:21.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:21.370+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:22.118 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:21.958+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:22.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:21.958+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:22.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:21.958+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:22.119 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:21.958+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:22.119 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:21.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:22.119 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:21.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:22.119 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:21 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:21.879+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:22.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:22.132+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12662 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:22.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:22.371+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:22.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:22.371+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:22.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:22.371+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:22.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:22.371+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:23.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:22.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:23.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:22.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:23.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:22.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:23.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:22.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:23.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:22.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:23.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:22.970+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:23.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:22 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:22.970+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:23.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:23.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:23.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:23.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:23.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:23.335+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:23.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:23.335+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:24.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:23.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:24.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:23.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:24.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:23.951+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:23.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:23.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:23.945+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:24.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:23 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:23.945+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:24.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:24.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:24.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:24.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:24.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:24.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:24.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:24.351+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:25.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:24.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:25.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:24.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:25.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:24.909+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:24.951+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:25.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:24.951+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:25.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:24.951+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:25.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:24 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:24.951+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:25.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:25.391+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:25.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:25.391+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:25.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:25.391+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:25.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:25.391+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:26.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:25.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:26.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:25.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:26.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:25.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:26.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:25.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:26.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:25.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:26.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:25.996+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:26.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:25 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:25.996+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:26.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:26.418+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:26.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:26.418+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:26.418+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:26.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:26.418+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:27.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:27.026+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:27.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:27.026+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:27.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:27.026+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:27.120 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:27.027+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:27.121 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:26.829+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:27.121 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:26.829+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:27.121 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:26 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:26.829+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:27.414 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:27.134+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12711 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:27.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:27.381+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:27.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:27.381+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:27.414 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:27.381+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:27.415 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:27.381+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:28.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:27.859+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:28.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:27.859+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:28.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:27 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:27.859+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:28.061+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:28.061+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:28.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:28.061+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:28.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:28.061+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:28.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:28.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:28.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:28.398+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:28.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:28.398+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:29.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:28.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:29.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:28.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:29.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:28 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:28.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:29.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:29.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:29.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:29.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:29.057+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:29.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:29.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:29.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:29.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:29.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:29.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:29.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:29.378+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:30.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:29.928+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:30.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:29.928+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:30.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:29 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:29.928+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:30.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:30.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:30.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:30.077+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:30.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:30.077+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:30.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:22:30.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:22:30.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:30.401+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:30.401+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:30.401+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:30.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:30.401+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:31.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:30.936+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:31.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:30.936+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:31.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:30 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:30.936+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:31.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:31.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:31.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:31.052+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:31.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:31.052+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:31.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:31.388+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:31.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:31.388+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:31.388+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:31.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:31.388+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:32.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:32.135+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12761 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:32.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:31.922+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:32.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:31.922+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:32.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:31 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:31.922+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:32.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:32.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:32.063+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:32.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:32.063+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:32.353+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:32.353+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:32.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:32.353+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:32.707 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:32.353+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:33.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:33.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:33.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:33.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:33.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:33.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:33.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:33.045+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:33.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:32.904+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:33.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:32.904+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:33.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:32 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:32.904+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:33.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:33.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:33.363+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:33.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:33.363+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:33.363+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:33.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:33.363+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:34.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:33.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:34.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:33.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:34.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:33 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:33.899+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:34.039+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:34.039+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:34.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:34.039+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:34.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:34.039+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:34.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:34.343+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:34.343+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:34.343+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:34.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:34.343+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:34.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:35.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:34.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:35.207 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:34 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:34.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:35.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:35.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:35.018+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:35.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:35.018+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:35.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:35.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:35.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:35.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:35.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:35.350+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:35.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:35.350+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:36.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:36.058+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:36.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:36.058+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:36.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:36.058+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:36.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:36.058+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:36.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:35.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:36.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:35.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:36.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:35 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:35.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:36.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:36.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:36.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:36.319+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:36.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:36.319+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:37.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:37.137+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12807 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:37.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:36.929+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:37.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:36.929+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:37.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:36 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:36.929+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:37.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:37.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:37.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:37.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:37.031+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:37.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:37.031+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:37.345+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:37.345+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:37.345+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:37.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:37.345+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:38.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:37.943+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:38.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:37.943+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:38.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:37 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:37.943+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:38.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:38.073+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:38.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:38.073+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:38.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:38.073+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:38.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:38.074+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:38.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:38.390+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:38.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:38.390+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:38.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:38.390+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:38.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:38.390+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:39.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:38.907+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:38.907+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:39.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:38 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:38.907+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:39.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:39.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:39.057+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:39.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:39.057+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:39.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:39.355+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:39.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:39.355+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:39.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:39.355+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:39.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:39.355+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:40.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:39.944+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:40.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:39.944+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:40.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:39 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:39.944+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:40.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:40.058+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:40.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:40.058+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:40.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:40.058+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:40.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:40.058+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:40.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:40.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:40.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:40.378+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:40.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:40.379+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:41.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:40.934+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:40.934+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:41.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:40 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:40.934+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:41.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:41.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:41.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:41.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:41.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:41.012+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:41.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:41.013+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:41.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:41.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:41.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:41.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:41.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:41.351+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:41.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:41.351+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:42.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:42.140+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12852 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:42.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:41.908+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:42.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:41.908+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:42.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:41 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:41.908+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:42.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:42.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:42.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:42.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:42.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:42.043+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:42.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:42.043+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:42.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:42.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:42.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:42.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:42.330+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:42.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:42.330+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:43.011 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:42.947+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:43.011 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:42.947+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:43.011 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:42 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:42.947+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:43.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.026+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:43.351 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.026+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:43.352 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.026+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:43.352 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.026+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:43.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:43.366+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:43.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:43.366+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:43.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:43.366+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:43.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:43.366+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:44.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.984+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:44.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.984+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:44.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.984+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:44.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:43.984+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:43.933+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:43.933+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:44.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:43 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:43.933+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:44.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:44.372+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:44.372+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:44.372+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:44.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:44.372+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:45.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:44.934+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:45.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:44.934+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:45.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:44.934+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:45.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:44.952+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:45.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:44.952+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:45.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:44.952+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:45.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:44 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:44.952+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:45.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:22:45.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:22:45.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:45.389+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:45.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:45.389+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:45.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:45.389+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:45.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:45.389+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:46.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:45.897+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:46.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:45.897+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:46.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:45.897+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:45.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:45.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:45.989+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:46.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:45 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:45.989+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:46.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:46.410+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:46.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:46.410+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:46.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:46.410+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:46.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:46.410+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:47.205 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:47.142+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12898 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:47.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:46.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:47.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:46.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:47.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:46 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:46.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:47.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:47.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:47.017+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:47.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:47.017+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:47.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:47.400+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:47.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:47.400+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:47.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:47.400+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:47.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:47.401+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:48.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:47.895+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:48.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:47.896+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:48.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:47 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:47.896+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:48.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:48.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:48.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:48.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:48.045+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:48.207 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:48.045+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:48.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:48.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:48.403+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:48.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:48.403+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:48.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:48.403+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:48.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:48.403+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:49.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:48.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:49.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:48.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:49.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:48 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:48.892+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:49.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:49.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:49.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:49.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:49.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:49.025+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:49.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:49.025+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:49.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:49.417+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:49.417+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:49.417+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:49.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:49.418+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:50.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:49.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:50.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:49.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:50.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:49 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:49.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:50.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:50.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:50.046+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:50.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:50.046+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:50.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:50.448+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:50.448+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:50.448+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:50.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:50.448+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:51.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:51.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:51.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:51.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:51.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:51.013+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:51.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:51.013+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:51.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:50.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:51.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:50.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:51.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:50 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:50.903+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:51.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-15T23:22:51.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T23:22:51.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-15T23:22:51.486 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-15T23:22:51.487 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi037 2024-04-15T23:22:51.503 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T23:22:51.517 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T23:22:51.568 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: timeout: 2024-04-15T23:22:51.568 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T23:22:51.601 DEBUG:teuthology.nuke:shortname: smithi037 2024-04-15T23:22:51.602 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T23:22:51.633 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:32:20.117607', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLZmM5gvh1qzwCEtC1kOi+EYXitmVn46BT27K1/Vstizybv9ZrD4lUjcg3tHtS5YT1WLD+qu0mAE3oovDFJN5I='} 2024-04-15T23:22:51.634 DEBUG:teuthology.nuke:shortname: smithi189 2024-04-15T23:22:51.635 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T23:22:51.657 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657608', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:32:20.119725', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP2CqhUmieuRAYfPqi7ZKgTMYFB2yekVdQVGUOjtELRJVw4kwAKHMbd0EWWVYnJxrFzXmmC2+2AAqANYA3B/FHc='} 2024-04-15T23:22:51.689 INFO:teuthology.orchestra.console.smithi037:Power off 2024-04-15T23:22:51.690 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T23:22:51.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:51.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:51.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:51.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:51.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:51.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:51.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:51.399+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:51.714 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2024-04-15T23:22:51.714 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:22:51.733 INFO:teuthology.orchestra.console.smithi189:Power off 2024-04-15T23:22:51.733 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T23:22:51.743 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-04-15T23:22:51.765 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-04-15T23:22:51.765 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:22:51.790 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-15T23:22:52.129 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:52.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:52.130 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:52.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:52.130 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:52.042+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:52.131 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:52.042+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:52.131 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:51.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:52.131 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:51.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:52.132 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:51 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:51.877+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:52.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-mon-smithi189[36175]: 2024-04-15T23:22:52.144+0000 7f2352328700 -1 mon.smithi189@1(probing) e2 get_health_metrics reporting 12945 slow ops, oldest is mdsbeacon(24301/cephfs.smithi189.kzokod up:standby fs=cephfs seq=33 v16) 2024-04-15T23:22:52.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:52.364+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:52.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:52.364+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:52.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:52.364+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:52.456 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:52.364+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:53.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.001+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:53.205 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.001+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:53.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.001+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:53.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.001+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:53.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:52.870+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:53.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:52.870+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:53.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:52 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:52.870+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:53.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:53.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:53.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:53.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:53.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:53.399+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:53.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:53.399+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:54.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:53.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:54.205 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:53.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:54.206 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:53.910+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:54.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:54.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:54.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.966+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:54.206 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:53 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:53.966+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:54.705 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:54.425+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:54.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:54.425+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:54.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:54.425+0000 7f83b4f4c700 -1 osd.4 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:43.079488+0000 (oldest deadline 2024-04-15T23:21:03.079488+0000) 2024-04-15T23:22:54.706 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-4[46923]: 2024-04-15T23:22:54.425+0000 7f83b4f4c700 -1 osd.4 46 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.1.8:149 2.11 2:8957603a:::201.00000007:head [write 2954273~3902 [fadvise_dontneed] in=3902b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:55.170 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:54.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:55.171 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:54.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:55.171 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-3[42225]: 2024-04-15T23:22:54.911+0000 7f9f42d4d700 -1 osd.3 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:35.251850+0000 (oldest deadline 2024-04-15T23:20:55.251850+0000) 2024-04-15T23:22:55.171 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:54.949+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6806 osd.0 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:55.171 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:54.949+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6814 osd.1 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:55.171 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:54.949+0000 7f1736a32700 -1 osd.5 46 heartbeat_check: no reply from 172.21.15.37:6822 osd.2 ever on either front or back, first ping sent 2024-04-15T23:20:25.768999+0000 (oldest deadline 2024-04-15T23:20:45.768999+0000) 2024-04-15T23:22:55.171 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 15 23:22:54 smithi189 ceph-60ff499a-fb7a-11ee-bc8f-c7b262605968-osd-5[51637]: 2024-04-15T23:22:54.949+0000 7f1736a32700 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.8:162 2.17 2:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-04-15T23:22:55.744 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:22:55.791 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:22:55.855 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-04-15T23:22:55.902 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-15T23:22:59.858 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:22:59.903 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:22:59.970 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-15T23:23:00.013 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-15T23:23:00.071 INFO:teuthology.orchestra.console.smithi037:Power off completed 2024-04-15T23:23:04.018 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T23:23:04.128 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-04-15T23:23:04.229 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-04-15T23:23:04.336 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2252.0192518234253 failure_reason: 'Command failed (workunit test suites/fsstress.sh) on smithi189 with status 126: ''mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=461a84e6fbbf089011eafd13b98169fb9e9b8cc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=75dd15f32ab747bcb0e47b9418dc6def status: fail success: false 2024-04-15T23:23:04.337 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T23:23:04.419 INFO:teuthology.run:FAIL